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SUBJECT:  Pershing'  II  Follow-On  Test:  Size  Reduced  by  Sequential 

Analysis 


By  memorandum  of  30  August  1982  (Reference  1),  the  Under 
Secretary  of  the  Army  tasked  the  service  to  "review  our 
[operational  test]  methodology,  to  include  considerations  of 
mathematical  rigor,  risks,  planning  horizon,  costs,  and 
operational  matters."  In  discussion  of  this  matter  with  the 
author,  he  further  elaborated  the  objectives: 

a)  Minimize  cost  of  testing  over  the  program  life.  Monitor 
all  test  results,  including  those  of  components  as  well  as  of  the 
system,  to  minimize  "no-tests"  and  to  save  on  full-up  tests.  Use 
sequential  analysis  to  further  pare  requirements  for  missile 
flights . 

b)  Criteria  of  test  adequacy  should  be  no  more  severe  than 
those  of  other  services  (e.g.,  Minuteman,  Poseidon). 

c)  Challenge  the  necessity  for  an  annual  update. 

d)  Consider  whether  testing,  maintenance  float,  and  reload 
were  independent  requirements  as  opposed  to  multiple  missions  for 
the  same  inventory  of  missiles. 

The  task  was  passed  to  the  Army  Research  Office  (Research 
Triangle,  NC)  which  manages  the  business  of  the  Army's  Mathematics 
Steering  Committee  (Dr.  Jagdish  Chandra,  Chairman),  supporting 
mathematical  research  of  relevance  to  the  Army  and  the 
improvements  in  mathematical  methods  employed  in  the  Army's 
research  and  study  agencies. 

The  work  summarized  here  is  composed  of  contributions  of 
several  statisticians  whose  aid  was  solicited  by  the  AMSC:  Dr. 
Michael  Woodroofe  (University  of  Michigan)*,  Dr.  Nozer 
Singpurwalla  (George  Washington  University),  and  Dr.  Robert  Launer 
(Army  Research  Office),  as  well  as  the  author  of  this  report. 
Others  have  provided  informal  comments  and  criticisms.  An  early 
version  of  this  paper,  prior  to  the  author's  knowledge  of  this 
other  research,  was  presented  as  a  talk  at  a  conference  of  Army 
mathematicians  (Reference  2). 


*  At  Rutgers  University  during  the  course  of  this  research. 


Chapter  I 
The  Problem 


Two  documents  combined  set  forth  the  guidance  the  Joint 
Chiefs  of  Staff  have  provided  to  the  military  services  regarding 
the  conduct  and  reporting  of  tests  of  certain  systems.  For  the 
Army  only  the  Pershing  Missile  system  is  covered  (Pershing  I  and 
la,  and  now  Pershing  II). 

In  a  memorandum  of  1975  (Reference  3),  the  Joint  Chief  of 
Staff  directed  that  numerical  confidence  statements  should  be 
based  on  WSEG  Report  92C  (Reference  4),  an  extract  of  which  is  at 
Appendix  C.  "The  goal  of  a  test  program  should  be  to  allow 
detection  of  a  minimum  change  of  X  percent  at  the  Y  percent 
confidence  level."  *  It  suggests,  by  way  of  example,  the  use  of 
Fisher's  Exact  Test  to  demonstrate  success  or  failure  in  meeting 
this  criterion. 

References  3  and  4  have  just  been  superseded.  The  revisions 
(References  5  and  6)  eliminate  an  ambiguity  and  add  considerations 
not  previously  called  for  and  not  discussed  here  except  to  note 
that  the  criteria  to  be  applied  to  Pershing  II  are  now  less 
demanding  than  those  applied  to  strategic  systems.  Fisher's  Exact 
Test  is  still  countenanced. 

This  use  of  this  criterion  appeared  to  the  author  to  lack  a 
sound  statistical  justification,  and  attempts  to  patch  it  up  were 
unsuccessful.  Appeal  to  a  number  of  practicing  statisticians 
within  and  outside  the  Army  supported  my  challenge  to  Fisher's 
Exact  Test  (FET)  in  its  application  to  Pershing  reliability 
tracking.  No  one  was  contesting  the  ability  of  the  FET  to  provide 
estimates  of  the  probability  that  two  samples,  which  have  yielded 
pass-fail  data,  come  from  the  same  parent  population,  though 
Kendall  and  Stuart  (Reference  7),  do  condemn  its  use  for  small 
samples . 

With  such  an  error  apparently  arising  from  an  application  of 
the  methods  of  the  "frequency"  school  of  statistics,  the  obvious 
alternative  was  to  try  the  methods  of  the  "Bayesian"  school. 

There  are  many  expositions  of  methods  based  on  the  use  of 
Bayes'  Theorem,  the  most  recent  of  which — "Bayesian  Reliability 
Analysis"  by  Martz  and  Waller — (Reference  8)  I  shall  quote  at 
intervals.  Among  the  works  arguing  for  the  adoption  of  Bayesian 
methods,  the  following  are  noteworthy: 


*  X  and  Y  are  classified  numbers. 


,  Raiffa  and  Schlaifer  -  Applied  Statistical  Decision  Theory 
(Reference  9)  with  a  very  complete  description  of  the  method  of 
conjugate  prior  distributions. 

Jaynes  E.T.,  "Prior  Probabilities"  (IEEE  Transactions  on 
System  Science  and  Cybernetics,  September  1968)  (Reference  10). 
Deduction  from  the  principles  of  maximum  entropy  and  invariance 
under  certain  group  transformations  leads  directly  to  the  Beta 
distribution  as  conjugate  prior  to  a  Bernoulli  process;  indeed  to 

where  s  is  the  number  of  successes  in  n  trials  observed  as  the 
basis  for  estimating  p.  This  removes  some  of  the  "ad  hoc"  or 
"mathematically  convenient"  color  of  conjugate  priors  when  relying 
on  Raiffa  and  Schlaifer. 

Martz  and  Waller  perhaps  epitomize  the  case  best: 

"There  are  several  benefits  in  using  Bayesian  methods  in 
reliability.  First  of  all,  it  is  important  to  recognize  that  all 
statistical  inferential  theories,  whether  sampling  theory, 
Bayesian,  likelihood,  or  otherwise,  require  some  degree  of 
subjectivity  in  their  use.  Sampling  theory  requires  assumptions 
concerning  such  things  as  a  sampling  model,  confidence 
coefficient,  which  estimator  to  use,  and  so  on.  For  example,  a 
sampling  theory  analysis  proceeds  as  if  it  were  believed  a  priori 
that  the  data  were  exactly  [exponentially]  distributed,  that  each 
observation  had  exactly  the  same  mean  life  0,  and  that  each 
observation  was  distributed  exactly  independently  of  every  other 
sample  observation.  The  Bayesian  method  provides  a  satisfactory 
way  of  explicitly  introducing  and  organizing  assumptions  regarding 
prior  knowledge  or  ignorance.  These  assumptions  lead  via  Bayes’ 
theorem  to  posterior  inferences,  that  is,  inference  obtained  once 
the  data  have  been  incorporated  into  the  analysis,  about  the 
reliability  parameter(s)  of  interest.  Bayes’  theorem  provides  a 
simple,  error-free  truism  for  incorporating  the  prior  information. 
The  engineering  judgment  and  prior  knowledge  are  brought  out  into 
the  open  and  are  there  for  everyone  to  see  instead  of  being 
quietly  hidden.  The  engineer  usually  appreciates  this  opportunity 
to  divulge  such  prior  information  in  a  formalized  way." 

The  authors  I  commend  are  not,  on  philosophical  matters,  in 
complete  agreement,  and  the  authors  (and  critics)  of  the  methods 
proposed  in  this  paper  have  their  differences,  some  of  which 
become  important  as  we  proceed. 


.  Suffice  it  to  say  that  the  Bayesian  approach  requires  a  more 
careful  statement  of  the  problem,  to  include  in  particular  the 
prior  distribution  function,  costs  and  risks:  matters  which  the 
frequentists  collapse  into  the  confidence  limits  and  ^  . 

If  there  is  indeed  a  legitimate  uncertainty  in  (the  form  of)  the 
prior  distribution,  that  uncertainty  must  surely  propagate  into  an 
uncertainty  in  the  predictions  for  the  process.  In  some  cases 
results  can  be  shown  to  be  insensitive  to  the  prior,  and  thus  a 
convergence  of  Bayesian  and  frequentist  answers  occurs;  but 
lacking  such  invariance,  the  frequentists  are  hard  pressed  to 
prove  they  have  solved  the  right  problem. 

Having  said  this,  I  must  confess  that  for  some  purposes  we 
shall  employ  the  frequentist  approach,  primarily  because  a  full 
Bayesian  solution  has  not  been  worked  out. 

Section  _1 .  Literal  Interpretation  of  JCS  Guidance: 

".  .  .  annual  .  .  .  detection  of  a  minimum  reliability  change 
of  X  percent  at  the  Y  percent  confidence  level." 

A  "change"  in  something  means  that  its  previous  value  has 
been  defined.  It  would  appear  that  an  evaluation  of  the  results 
of  the  first  year's  Follow-on-Test  (FOT)  is  to  be  compared  to  that 
of  the  Operational  Test  (the  base-line ) (OT ) ,  and  the  evaluations 
of  subsequent  FOTs  are  to  be  compared  to  the  evaluations  made  a 
year  ago.  The  tests  being  of  something  less  than  the  full  combat 
mode  of  the  system,  projection  to  combat  capability  is  to  be  made; 
thus  while  test  results  are  to  be  reported,  they  are  to  be 
interpreted  as  well.  This  interpretation  is  surely  to  be  based  on 
all  prior  knowledge  of  system  performance;  i.e.,  all  prior  testing 
as  well  as  that  most  recently  at  hand,  "weighted"  (one  might  say) 
by  expert  judgment  of  the  relevance  of  older  tests  and  analysis. 

In  the  case  of  Pershing  II,  we  shall  have  an  inventory  of 
missiles  produced  over  a  period  of  time  and  expected  to  be  in 
service  for  a  longer  period.  From  the  point  of  view  of 
homogeneity,  the  inventory  may  need  to  be  divided  into  two  or  more 
blocks,  based  on  the  significance  of  any  changes  in  the  production 
process  during  the  run.  When  they  are  subjected  to  (annual)  test, 
missiles  will  be  of  different  ages  as  well  from  different  blocks; 
so  serial  number  and  age  may  influence  reliability  at  the  time  of 
testing  or  use  in  combat.  It  is  clear,  then,  that  in  treating  of 
a  "change"  in  reliability,  we  are  dealing  with  an  uncertain  base. 
Options  which  are  open  to  us  include: 

a)  Computing  a  "best"  estimate  from  the  OT  firings,  and 
treating  it  as  the  exact  value  of  the  reliability  at  that  time  of 
all  the  inventory. 


.  b)  Computing  as  in  (a),  but  associating  an  uncertainty 
(standard  deviation)  to  it  also,  to  describe  the  uncertain 
reference  point. 

In  either  case,  the  results  of  each  subsequent  (annual)  test 
would  be  compared  to  this  as  standard. 

c)  Computing  as  in  (b),  but  then  modifying  the  estimates 
using  the  results  of  subsequent  tests  (more  trials,  more 
successes,  more  failures).  There  are  extremes  in  this  process 
which  are  to  be  avoided: 

(i)  This  modification  might  consist  of  using  only  the 
previous  year's  results  as  indication  of  the  remaining  inventory. 

(ii)  This  modification  might  consist  of  accumulating 
the  results  of  all  prior  tests,  without  regard  to  the  aging  effect 
or  block  modifications. 

Judgment  is  clearly  needed.  Limiting  the  criterion  to  the 
smallness  of  the  latest  annual  change  (with  small  samples  in  the 
two  cases)  could  result  in  a  dangerous  accumulation  of  change  over 
the  system  life.  On  the  other  hand,  where  no  statistically 
significant  change  has  been  detected,  it  would  be  reasonable  to 
add  one  year's  results  to  the  results  of  the  whole  prior  test 
series  of  a  homogeneous  block  in  estimating  the  average  value  at, 
say,  the  average  age  of  the  tested  articles.  It  is  probably  not 
possible  to  specify  in  advance  the  details  of  the  critical  results 
to  be  reported.  What  is  more  important  is  that  analyses  be 
conducted  to  discover  what  are  the  constant  and  what  are  the 
variable  components  of  the  system  reliability.  Finally,  detection 
of  a  trend  should  make  it  possible  to  forecast  when  the  results  of 
that  trend  will  no  longer  be  tolerable,  and  so  signal  the  degree 
of  urgency  with  which  management  should  act  to  correct  the  trend. 

d)  This  brings  us  to  the  question  of  the  frequency  of 
reporting  the  results  of  testing  and  analysis.  The  current 
practice  is  an  annual  report  which  probably  has  its  roots  in 
adminstrative  cycles.  The  technical  problems  which  reporting 
communicates  to  management  are  probably  of  two  sorts:  long-term 
aging  with  gradual  deterioration,  ("one-hoss  shay"  syndrome)  and 
catastrophic  failures.  The  latter  tend  to  announce  their  presence 
in  consistent  repetitions  of  particular  failure  modes,  and  so  call 
for  out-of-cycle  action  no  matter  what  the  standard  interval 
between  reports.  The  former,  on  the  other  hand,  are  evidence  of 
problems  only  slowly  exacerbating,  and  so  allow  a  more  leisurely 
pace  of  administrative  response.  Alternatives  to  the  present 
annual  cycle  are  proposed  below,  for  situations  in  which  no 
guarantee  of  a  clear  bright  green  light  or  red  light  is  available 
annually:  (i)  A  guarantee  can  be  given  of  a  low  likelihood  of 
having  to  wait  more  than,  say,  16  months  for  such  a  signal,  along 
with  the  provision  of  a  technical  review  of  all  failures  showing 
any  repetitions  of  mode.  (ii)  Administratively,  skipping  one 
year's  report  may  be  simpler. 


.  These  options  will  be  explored  in  one  or  more  places  in  the 
mathematical  sections  to  follow. 

Two  assumptions  have  immediately  to  be  disposed  of: 

1)  Because  Fisher's  Exact  Test  is  mentioned  in  JCS 
guidance,  its  use  is  correct  and  mandatory. 

Fisher's  Exact  Test  is  an  enumeration  of  all  possible 
relative  outcomes  in  two  series  of  pass-fail  tests,  subject  to  the 
restraints  that  the  numbers  of  tests  in  each  series  be  fixed  and 
the  combined  number  of  successes  also.  It  yields  the  probability 
that  the  articles  tested  in  the  two  series  were  drawn  from  the 
same  population — one  with  a  fixed  probability  of  pass.  If  the 
total  number  of  successes  is  not  controlled,  the  results  of 
FET  admit  of  this  interpretation  only  in  the  limit  of  large 
samples.  Given  that  the  probability  of  success  could  be  different 
in  the  two  populations,  it  is  sometimes  claimed  that  FET  can  be 
used  to  estimate  the  probability  that  they  differ  by  prescribed 
amounts.  This  claim  is  unwarranted.  The  JCS  could  be  faulted  for 
suggesting  the  test,  but  they  did  not  underwrite  the  extended  use 
as  in  the  Army's  methodology.  (See  Kendall  and  Stuart;  also 
Chapter  III ) . 

2)  We  can  know  the  reliability  of  an  object. 

We  shall  never  know  the  "true”  as-manufactured  reliability  of 
the  components  of  the  Pershing  system,  and  much  of  such  knowledge 
as  we  do  gain  will  come  at  the  expense  of  tactical  inventory.  It 
may  be  that,  for  the  purposes  of  designing  tests  of  operational 
reliability,  we  need  not  know  this  a  priori  probability  with  any 
great  accuracy;  and  so  methods  which  treat  it  as  known  for  this 
purpose  may  be  satisfactory.  This  does  not  justify  the  assumption 
when  analyzing  the  results  of  actual  tests. 


Section  2  .  Mat hem atical  Preliminaries 


Mayes'  Theorem:  The  Need  for  a  Prior  Distribution 

Essential  to  much  of  what  follows  is  Bayes'  Theorem,  sketched 
here  as  background.  The  conditional  probability  of  an  event  B, 
given  that  another  event  A  has  occurred,  is  symbolized  and  defined 


■p  (b  i 


y 

V(A) 


where  P(A)  (•=£())  is  the  marginal  probability  of  event  A,  and 
P(A,B)  is  the  probability  of  joint  occurrence  of  A  and  B.  One  may 
also  speak  of  P(A/B)  =  P(A,B)/P(B)  with  similar  meanings  and 
1 imits,  leading  to 

PC*)  *  PO(^  Pte)  n 

Given  that  B  can  occur  in  n  ways  Bi  ( i=l , 2 , . . . , n )  one  of  which 
always  occurs  with  A,  we  may  sum  expressions  like  Eq.  1.3  for  the 
entire  set  of  events  Bi 

•PCrtXj  'Pfeil(\)  =  '2:;'P(AISi'>P(8iWP(:M  1.4 

os  the  multiplier  of  P(A)  is  equal  to  1,  having  encompassed  all 
possible  pairings.  If  P(A)  ^  0,  we  have  Bayes'  Theorem: 


PCr.iaV  P(aIbA'P(6;) 


Suppose  now  that  events  Bi  are  logically  (causally)  prior  to 
event  A.  Then  P(Bi)  is  called  the  prior  distribution  of 
Bi,  P( A/Bi )  the  likelihood  of  A,  given  Bi,  P(A)  the  marginal 
distribution  of  A,  and  P(Bi/A)  the  posterior  distribution  of  Bi . 
Bayes'  Theorem,  given  in  symbols  by  Eq.1.5,  may  then  be  stated  in 
words : 


Posterior  Distribution 


F.ior  Distribution  X  Likelihood  (Function 
Marginal  Distribution 


(This  argument  holds  for  both  discrete  and  continuous  distributions 
of  probability.) 

Likelihood  functions  are  a  familiar  staple  of  probability 
theory,  being  forecasts  of  the  frequency  of  chance  events  A  based  or. 
presumptions  about  certain  prior  events  or  conditions  (a  die  that  is 
unbiased,  the  "normal"  distribution  of  errors,  half-life  of  a  know: 
radioactive  substance).  Marginal  distributions  then  are  forecasts  of 


the  .results  of  experiments.  Bayes'  Theorem  tells  us  that  inferences 
about  the  events  Bi  which  lead  to  a  marginal  distribution  cannot  be 
derived  from  the  likelihood  function  alone,  but  require  knowledge  of 
the  prior  distribution  P(Bi)  as  well.  In  the  context  of  our  task,  we 
need  to  know  more  than  the  results  of  a  set  of  missile  firings  to 
infer  the  reliability  of  the  missile. 

Other  requirements  of  a  Bayesian  analysis  will  be  discussed  as 
the  issues  arise. 

Section  3^.  Illustration  of  an  Analysis  in  Accord  with  JCS 
Guidelines 


We  assume  that  the  missiles  and  associated  ground  equipment  used 
in  an  annual  test  do  come  from  a  homogeneous  population,  and  that  the 
several  tests  within  that  year  are  statistically  independent.  We 
assume  further  that  the  reliability  p  is  definable,  and  then  may 
assert  that  were  we  to  know  p,  the  probability  of  s/  successes  and  f 
failures  in  n,'  trials  (nl'  =  si'  +  f 1 ' )  would  be  by  Bernoulli's 
formula  (a  likelihood  function): 


From  component  testing,  comparison  with  similar  systems, 
comparison  with  other  products  of  the  same  manufacturer,  engineering 
analysis,  we  should  develop  an  estimate  of  p  and  a  measure  of  our 
confidence  in  that  estimate.  Methods  exist,  e.g.  that  of  Maximum 
Entropy  (Reference  10),  for  constructing  from  this  information  a 
function  with  the  properties  of  a  probability  distribution — a  prior 
distribution.  Constraints  of  reasonableness  and  mathematical 
convenience  come  into  the  selection  process.  With  limited 
information  at  hand,  there  may  be  no  unique  solution.  The  analyst  is 
free  to  try  several  priors  and  to  observe  the  sensitivity  of  answers 
to  such  variations. 


Given  a  likelihood  function,  there  can  generally  be  found  a 
"conjugate"  prior  function  (so-called  because  it  marries 
mathematically  to  the  likelihood  function);  properly  a  class  of  such 
functions,  dependent  on  a  limited  number  of  parameters  to  distinguish 
members  of  the  class.  Conjugate  to  the  Bernoulli's  distribution  is 
the  Beta  distribution,  written 
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Different  sets  of  the  parameters  s0  and  f0  give  rise  to  functions 
whose  graphs  are  variously  peaked  at  some  locale  within  the  limits  of 
0  to  1  ,  are  relatively  flat,  are  J-shaped  and  strongly  peaked  at  0  or 
1,  or  are  even  U-shaped  and  strongly  peaked  at  both  0  and  1.  It  is  a 
rich  set  of  functions. 


get 


Taking  the  product  of  dP(s0,f0)  with  the  Bernoulli  function,  we 
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which  when  integrated  over  the  range  of  0  to  1  gives 
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the  marginal  distribution  of  s,'  given  B(s0,f0)  as  prior.  The  ratio 
of  Eqs.  1.6  and  1.7  gives  the  posterior  distribution  of  p  for  s, 1  and 
f  \  '  observed: 
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explaining  my  notation  and  revealing  the  meaning  of  conjugation. 

From  a  prior  distribution  B(s0,f0),  and  a  likelihood  function 
for  a  test  of  a  sample  of  size  n,',  we  have  created  a  function  which, 
as  a  posterior  distribution  from  that  experiment,  is  logically  the 
prior  when  testing  a  second  sample  of  size  n^' .  This  process  can  be 
repeated  ad  libitum,  making  sample  1  refer  to  all  prior  information 
and  sample  2  the  latest  test. 


No-w  the  JCS  asks  to  know  the  probability  that  the  reliability  of 
sample  2  (and  by  inference  that  of  the  population  from  which  it  was 
drawn)  is  less  than  a  certain  fraction  k  ( o  <  k  l)of  the 
reliability  estimate  p  of  sample  1.  If  the  evidentiary  basis  for 
this  answer  lies  entirely  in  the  test  of  n2'  items,  then  we  may 
assume  instead  a  uniform  prior  distribution,  drop  the  primes  on  n2’, 
s2',  and  f  2  *  and  represent  this  probability  by 
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which  we  then  integrate  over  the  distribution  of  pi  to  get  the 
probability  that  p2  kpl: 
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The  probability  that  p 2 > k p 1  is  just  1  minus  this  result. 


As  an  aid  to  understanding  the  generality  of  this  result, 
consider  the  case  where  pi  =  rl  x  r3  and  p2*r2  x  r3  where  r3  is  a 
reliability  factor  not  subject  to  degradation  but  just  as  much 
subject  to  discovery  as  rl  and  r2.  Within  the  framework  of  Beta- 
function  priors,  we  might  be  led  to  the  posterior  distribution: 

-  Ky\S'~'  (i-**,)  '  ((- '  f*'  ’(V  1  \.\0 

where  s3(f3)  is  the  total  number  of  observed  successes  (failures)  of 
the  subsystems  described  by  r3.  For  any  values  of  r3  and  k  between  0 

and  1,  P(p2  kpl)  «=  P(r2  <  krl).  When  the  latter  function  is  given 

by  integrating  Eq.1.10  first  over  r3  from  0  to  1,  it  is  clear  that 
the  result  is  the  same  as  though  r3  *=  1  (i.e.,  it  can  be  ignored). 
Thus  using  the  criterion  p2  kpl  we  cam  be  freed  of  any  concern 
about  reliability  factors  common  to  pi  and  p2.  I  would  assert  that 
this  is  a  good  reason  to  employ  this  criterion  in  preference  to  the 
one  described  next. 

The  JCS  guidance  has  not  always  been  interpreted  as  speaking  to 
a  proportional  reduction  in  reliability;  sometimes  it  has  been 

interpreted  a:  measuring  a  reduction  of,  say,  lOOd  percentage  points* 


Instead  of  Eq.  1.9  we  would  then  use 


(While  we  have  strayed  from  the  neatness  of  conjugate  functions,  by 
reason  of  the  incomplete  integrals,  we  still  have  a  consistent 
method.  Similar  expressions  will  be  found  in  Reference  8,  p.  271.) 


*  Indeed,  the  latest  revision  of  the  JCS  guidance  (Reference  5) 
mandates  this  form  of  the  criterion. 


•  Eqs.  1.9  and  1.11  give  mathematical  meaning  to  the  JCS  guidance. 
If  at  the  chosen  confidence  level  it  is  deemed  that  there  has  been  no 
significant  change  in  the  reliability  between  samples  1  and  2,  then 
sample  2  should  be  merged  with  sample  1  in  preparation  for  the  next 
year's  testing.  Other  criteria  should  be  examined  also  (e.g., 
probability  that  there  has  been  no  significant  departure  from  a 
nominal  value),  but  that  does  not  refute  the  translation  into 
mathematics  of  the  JCS  guidelines. 

At  this  point  I  note  that  much  of  the  historical  course  of 
development  of  mathematics  has  been  devoted  to  a  search  for  solutions 
requiring  a  minimum  of  actual  manipulation  of  numbers.  The 
approximations  used  by  statisticians  are  simply  good  examples  of 
this.  The  ready  availability  today  of  powerful  computers  reduces  the 
need  to  employ  approximations  which  may  be  questionable  in  particular 
cases.  Most  of  the  calculations  to  be  described  here  have  been 
carried  out  on  a  programmable  hand  calculator  (HP-41)  or  home 
computer  (Apple,  Commodore,  etc.).  Accordingly,  the  reader  need  not 
be  concerned  with  an  apparent  intractibility  of  the  formulas.  They 
could  be  evaluated  in  the  field  by  the  troops  of  a  Pershing  fire 
un  i  t . 


There  are  two  matters  of  concern:  the  prior  distribution  and 
limits  to  the  size  of  Sample  1.  I  have  already  discussed  problems 
with  the  prior  distribution.  One  assertion  made  is  that  with 
increase  in  the  size  of  the  data  base  it  can  become  misleadingly 
narrow,  ignoring  "unknown-unknowns."  A  different  way  of  saying  this 
is  that  tests  performed  sufficiently  long  ago  may  be  irrelevant  in 
describing  the  present  state  of  the  missile  inventory;  the  meaning 
of  this  argument  is  that  a  larger  annual  test  size  is  needed  to 
compensate  for  stale  data  in  Sample  1.  The  question  of  test  size 
will  be  the  subject  of  the  following  chapters.  Of  course,  if  there 
is  no  evidence  of  a  change  in  reliability  over  the  years,  there  is  no 
reason  to  purge  old  data. 

Section  _4.  Optimum  Test  Size 

In  order  to  determine  the  number  of  missiles  which  must  be 
procured  in  the  next  few  years  to  support  a  test  program  through  a 
long  period  of  service  life,  one  must  have  an  estimate  of  the  average 
annual  consumption  in  testing.  To  get  this  estimate,  especially  if 
it  be  glorified  by  a  phase  like  "optimum  test  size,"  one  must  know 
what  questions  the  tests  are  supposed  to  answer  and  how  frequently. 
This  in  turn  means  "getting  into  the  skull"  of  the  JCS.  We  must 
assume  that  first  of  all  there  is  sufficient  reason  to  conduct  the 
tests,  even  at  the  risk  of  compromise  of  properly-classified 
information.  We  know  that  there  will  be  a  finite  inventory,  and  that 
testing  reduces  that  inventory,  whether  or  not  it  be  formally  divided 
into  tactical  and  non-tactical  portions.  We  can  then  ask  the 


question:  how  does  the  result  of  an  additional  test  change  our 

perception  of  the  system  reliability,  and  so  of  the  sufficiency  of 
the  lesser  inventory  of  missiles  to  conduct  a  military  mission  should 
it  be  committed  to  combat  at  a  future  date?  Possible  answers  are 
discussed  in  Chapter  V.  As  there  are  circumstances  under  which  the 
answer  is  insensitive  to  the  size  of  the  inventory,  we  shall  spend 
more  time  considering  the  case  where  inventory  for  test  has  no 
tactical  mission. 

A  long  string  of  heads  or  tails  when  flipping  pennies  is  not 
impossible  or  even  incredible;  but  after  some  number,  one  is  entitled 
to  wonder  if  the  coin  is  biased.  Similarly,  when  testing  a  missile 
which  is  alleged  to  have  high  reliability,  a  string  of  failures — even 
a  short  one — challenges  the  presumption;  contrariwise,  a  long  string 
of  successes  tends  to  be  uninformative.  In  either  case  there  is  a 
practical  limit  to  the  value  of  the  additional  information  in  an 
outcome  merely  extending  such  a  string. 


To  address  this  problem  we  shall  invoke  the  discipline  of 
Sequential  Analysis,  to  include  Sequential  Probability  Ratio  Tests 
and  test  series  truncation.  Much  of  this  is  "old  hat",  having  been 
developed  in  World  War  II,  most  notably  by  Abraham  Wald  (Reference 
11)  working  on  military  problems,  and  largely  standardized  by  now. 
It  has  recently  been  reported  that  the  methods  were  independently 
developed  simultaneously  by  Alan  Turing  while  working  at  Bletchley 
Hall  to  crack  the  German  ENIGMA  codes  (Reference  12).  More 
importantly  there  is  recent  substantive  new  work  not  yet  "codified" 
in  text  books.  Two  applications  of  sequential  analysis  to  the 
Pershing  missile  test  problem  will  be  presented:  one  by  Nozer 
Singpurwalla  and  Robert  Launer  (Chapter  III)  and  one  by  Michael 
Woodroofe  (Chapter  IV).  While  aspects  of  the  treatment  will  appear 
more  "f requentist"  than  Bayesian,  both  evolve  into  completely 
Bayesian  solutions.  In  this  paper  I  shall  extract  from  their  work, 
and  comment  on  it  as  appropriate.  The  author  of  this  memorandum  is 
not  by  profession  a  statistician,  and  so  requests  that  the  original 
researchers  not  be  blamed  for  errors  in  translating  their  work  into 
this  format. 


Chapter  III 


Launer  and  Singpurwalla ' s  Proposal 

The  following  submission  by  Launer  and  Singpurwalla  is  the 
product  of  over  a  year  of  research  by  the  authors,  initiated  and 
guided  in  discussions  with  the  writer  of  this  note.  I  believe  it 
successfully  addresses  the  problem  placed  before  the  authors.  Note 
that  all  the  appendices  to  this  article  are  to  be  found  at  Appendix 
E. 


As  the  numerical  example  in  the  following  exposition  employs 
fictitious  data  and  arbitrary  values  of  the  parameters  ©C  * 
the  numerical  results  should  not  be  taken  as  applicable  to  the 
Pershing  II  problem.  The  dependencies  and  the  savings  from 
sequential  analysis  are  however  clearly  indicated,  the  penalty  when 
tests  are  batched,  and  the  potential  for  squeezing  information  out  of 
small  samples.  The  next  chapter  reports  further  steps  toward  savings 
through  careful  test  design. 


MONITORING  THE  RELIABILITY  OF  PERSHING  II  MISSILES— 

A  CRITIQUE  OF  THE  CURRENT  METHODOLOGY  AND  A  SUGGESTED 
COMBINED  BAYES IAN-SAMPLE  THEORETIC  APPROACH  + 

by  • 

Robert  Launer* 

Nozer  D.  Singpurvalla** 

1.  INTRODUCTION,  TEST  REQUIREMENTS,  AND  ASSUMPTIONS 

The  reliability  of  the  Pershing  II  missile  arsenal  is  an  unknown 
parameter  which  presumably  could  change  over  time.  To  monitor  the  re¬ 
liability,  and  also  to  ascertain  the  amount  of  change  in  reliability, 
if  any,  a  sample  of  n  Pershing  II  missiles  is  chosen  from  the  ar¬ 
senal  every  year,  and  each  missile  fired  to  observe  its  success  or  fail¬ 
ure.  The  testing  is  destructive,  and  the  arsenal  inventory  is  not 
replenished.  Thus,  it  is  highly  desirable  to  reduce  the  number  of  test 
missiles  fired  year  after  year.  Also,  if  possible,  it  is  desirable  to 
have  the  total  number  of  missiles  fired' per  year  be  a  multiple  of  three — 
that  is,  3,  6,  9,  etc.  A  stated  requirement  with  respect  to  the  year  by 
year  detection  of  change  in  reliability  is  that  a  change  of  &  should 
he  detected  with  a  probability  of  tt  or  more.  Since  the  test  data  are 

+  The  authors'  appendices  are  incorporated  in  this  paper  as  Appendix  E.  DW 
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of  a  pass-fail  nature,  a  correct  probability  model  for  describing  them 
is  the  binomial. 

Our  goal  is  to  determine  a  sample  size  and  a  decision  criterion 
that  will  satisfy  the  above  requirement,  and  minimize  the  total  amount 
of  testing.  Since  each  missile  is  expensive  to  produce  and  test,  there 
is  a  keen  desire  to  incorporate  into  the  analysis  all  knowledge  that  is 

available,  both,  from  the  previous  tests  and  engineering  experience. 
Thus  a  Bayesian  point  of  view  is  natural  here. 

2.  CRITIQUE  OF  PRESENT  METHODOLOGY 

% 

Based  on  our  reading  of  the  pertinent  literature  that  has  been 
made  available  to  us,  and  our  discussions  with  several  people  familiar 
with  the  test,  it  is  our  understanding  that  the  current  methodology  for 
analyzing  the  Pershing  II  data  is  based  on  Fisher's  exact  test,  hence¬ 
forth  FET.  We  claim  that  this  technique  is  inappropriate  for  the  situa¬ 
tion  described  above.  Furthermore,  a  modified  version  of  the  FET  which 
has  been  used  in  similar  situations  is  not  appropriate,  either.  Whereas 
the  FET  can  be  used  to  detect  the  equality  or  otherwise  of  two  binomial 
populations,  it  is  not  designed  to  detect  a  specified  difference  between 
the  two  binomial  parameters  in  question.  Furthermore,  FET  does  not  ad- 
dress  the  key  question  of  sample  size  selection,  and  thus  fails  to  ans¬ 
wer  the  main  question  posed  by  our  problem.  A  choice  of  the  sample  size 
should  be  based  on  an  assumed  or  target  value  of  the  reliability,  and 
this  is  nowhere  apparent  in  the  test. 

Given  a  sample  size  and  the  test  results  from  this  sample,  the 
FET  can  give  us  the  "p  values"  for  deciding  upon  the  difference  or 
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otherwise  of  the  two  binomial  populations  in  question,  and  this  may  be 
the  sole  motivation  for  using  this  test  here. 

3.  THE  COMBINED  BAYES I AN- S AMPLE  THEORETIC 
APPROACH  PROPOSED  HERE 

Our  proposed  approach  addresses  the  issues  posed  before,  and 
attempts  to  do  this  in  an  economical  manner  with  respect  to  sample  size. 

Since  reliability  changes  over  time,  we  introduce  an  index  t  , 

where  t  =  1,2,...  ;  thus  t  =  1  denotes  the  first  year  of  testing, 

t  =  2  denotes  the  second  year  of  testing,  and  so  on.  Let  nfc  denote 

the  number  of  missiles  to  be  tested  in  time  period  t  ;  nt  is  the 

(unknown)  sample  si2e,  one  of  our  decision  variables.  Let  x^  denote 

the  number  of  missiles  that  fire  successfully  in  time  period  t  ; 

note  that  0  <  x^  <  n 
t  t 

Let  p^  be  the  chance  that  any  missile  fired  at  t  will  fire 
successfully,  or  its  propensity  to  do  so.  Since  pt  is  unknown  to  us, 
we  express  our  uncertainty  about  it  by  a  probability  distribution,  say 
g(pt  |  previous  failure  data,  if  any,  and  H)  .  Thus  p£  is  treated  as 
an  unknown  parameter,  and  the  vertical  line  in  g(*)  denotes  conditioned 
upon  or  given,  and  H  denotes  our  background  information  about  p^  . 

If  we  have  no  previous  failure  data,  then  g(pt  |  H)  denotes  our  prior 
distribution  for  p  ;  otherwise  g(*  |  •)  denotes  our  posterior 
distribution. 

If  for  each  time  period  t  we  judge  the  missiles  in  the  arsenal 
to  be  exchangeable  (we  have  here  finite  exchangeability),  then  it  is 
appropriate  to  assume  that  given  p£  ,  the  probability  of  observing  x 


u 


successful  firings  in  a  sample  of  size  n  is  a  binomial  distribution; 


L 


that  is, 

P{xt  successes  in  firings  |  p^}  = 


'n  ' 

t 


PtC  (1  -  Pt) 


v\ 


(1) 


The  choice  of  the  sample  size  n  is  based  on  the  following  sample 
theoretic  arguments  for  testing  hypotheses  about  p^  . 

If  p  ,  the  chance  that  a  missile  is  fired  successfully  at  time 
t  ,  is  large,  then  the  number  of  failures  in  a  sample  of  size  nt  would 
tend  to  be  small.  Given  an  n^  and  having  specified  a  pt  ,  let  x* 
be  the  largest  integer  for  which  the  chance  of  observing  x*  or  fewer 
successes  is  small,  say  a  ;  that  is. 


x* 


Pix*  or  fewer  successes  in  n  |  p  }  =  £ 

C  C  j=0 


n 

t 

.  j  . 


(i  “  Pt)  <  a  • 
(2) 


be 


If  were  to  change  to  p^  -  A  ,  with  A  large,  then  the  num- 

r  of  failures  in  a  sample  of  size  would  tend  to  be  large;  if  A 


were  small,  the  number  of  failures  in  n^  would  tend  to  be  small.  Thus, 
for  some  small  number  8  , 

P{x*  or  fewer  successes  in  n^  firings  j  (p^  -  A) } 


l 

3=0 


n 


U  J 


n -j 


(3) 


C|  (Pt  -  A)J  (1  -  pt  +  A)  C  >  1  -  B  . 


If  in  (2)  and  (3)  we  assume  that  p£  ,  a  ,  8  ,  and  A  are  the 
only  known  quantities,  then  (2)  and  (3)  can  be  simultaneously  solved  to 
obtain  an  n^  and  x*  .  Once  this  is  done,  (2)  can  be  used  to  test  the 
null  hypothesis  that  the  reliability  of  the  missile  arsenal  at  time  t 


5 


is  p  ,  with  a  Type  I  error  a  •  This  is  done  by  accepting  (rejecting) 
the  null  hypothesis  whenever  x^.  >(^)  x*  ,  where  x^_  is  the  total  number 
of  successfully  fired  missiles  in  a  sample  of  size  n  .  If  a  =  .25 
and  B  -  -25,  then  (3)  assures  us  that  n^  and  x*  are  suitable  for 
detecting  the  desired  changes  in  reliability.  Note  that  (3)  describes 
the  power  of  the  test  as  specified  by  (2),  for  changing  values  of  A  . 

If  the  null  hypothesis  is  accepted,  we  conclude  that  the  reliability  of 

the  missile  arsenal  at  time  t  is  p 

*t 

In  our  case  p^_  is  not  specified,  as  it  is  an  unknown  parameter 
which  is  liable  to  change  over  time.  What  we  have  instead  is 
i.  a  prior  distribution  for  at  time  (t-1)  ,  say 

S  (Pt  I  (^2fX2^  *  *  *  * 9  ^^t~l  ,^t*“l^>  ^0  *  ^  ^  2  and 

g(p1  I  W)  ; 


ii.  a  posterior  distribution  for  pt  at  time  t  ,  say 
g(Pt  I  (n^x^,  ....  (nc>xc),  H)  ,  for  t  ^  1  . 
Thus,  if  we  uncondition  on  ,  (2)  and  (3)  would  become 

"  -j 


1  X* 

f  l 

r"t 

l  3=0 

13  . 

^  2  ,  and 

1  X* 

f  l 

n 

t 

l  I*0 

1  X* 

f  l 

* 

n 

t 

{ 

.  3  . 

i  t 

P^U  —  p^)  g(p^  |  •  •  .  ,  (n^_ p »  ^0  dp^  ^  tt  » 


Pt(l-Pt)  C  g Cp1  I  H) dp1  <  a  ,  for  t  =  1  ;  (A) 


(Pt~A):)  (1-Pt+A)  1  g(Pt  |  (n1,x1),...,(nt_1,xt_1),  H)dpt 


^  1  -  8  ,  for  t  >  2  , 


6 


1  x* 


/t  n  .  n  -j 

I  .  1  (p  -A)3 (1-p  +A)  C  g(p  |  K)dp 

j 


>  1  -  B  ,  for  t  =  1  . 


In  order  to  obtain  the  pair  (n^.x*)  ,  for  t  >  1  ,  we  need  to 
solve  (4)  and  (5)  sinul taneously .  Note  that  a  solution  to  (4)  and  (5) 
would  depend  on  our  choice  of  g(p^  |  •)  .  If  for  example,  g(p^  j  •)  is 
a  member  of  the  family  of  beta  density  functions,  then  (4)  and  (5)  would 
involve  incomplete  beta  functions  and  would  call  for  numerical  methods 
for  solving  them.  A  method  for  undertaking  this  is  described  in  Appendix 
A.  A  computer  code  for  implementing  the  method  of  Appendix  A  is  given  in 
Appendix  B.  An  example  using  the  above  is  in  Section  5. 

As  an  alternative  to  the  above,  and  one  which  is  easy  to  imple¬ 
ment,  we  replace  pt  in  (I)  and  (3)  by  pt  ,  the  modal  value  of 
g(Pt  I  (n^,x^),  .  ..,  (n  ^ ,xt-l^ »  ^  •  The  value  is  the  most 

likely  value  of  p^_  ,  given  all  the  previous  data,  and  is  determined  by 
the  prior  distribution  g(pt  |  (n^x^,  (n^.x^),  K)  .  The 

posterior  distribution  g(pt  |  (n^x^,  ...,  (nt,xt>,  »7)  represents  our 
best  assessment  of  the  arsenal  reliability  at  time  t  ,  given  all  the 
data  up  to  and  including  that  obtained  at  t  .  Its  model  value 
could  be  used  as  a  single  number  which  describes  p^  .  In  the  next  sec¬ 
tion,  we  discuss  an  implementation  of  the  above  alternative  procedure. 

An  implementation  of  the  main  procedure  follows  along  similar  lines, 
with  the  exception  that  in  computing  the  pair  Pt  is  not 

replaced  by  the  modal  value  of  its  prior  distribution. 
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3 . 1  Assessing  Our  Uncertainty  about  p _ and  Procedure  Implementation 

Since  p^  can  take  values  between  0  and  1,  a  convenient  but 
flexible  way  for  us  to  express  our  uncertainty  about  p  is  via  the 
family  of  beta  density  functions  on  (0,1).  Thus, 

1.  We  start  off  our  assessment  and  monitoring  procedure  by 
assigning  a  prior  distribution  for  p^  ,  say  g(p^  [  y,6,H)  , 
which  for  the  two  unknown  parameters  y  >  0  and  6  >  0  is 
a  beta  density  function 

g(Pl  1  Y.6.H)  =  f  (yK(6)  pI'1  (1-Pi)6~1  »  0  <  Pl  <  1  .  (6) 

The  modal  value  of  the  above  density  is 

-  _  3T-1 

pi  y+&-2  * 

Clearly,  p^  best  describes  in  the  form  of  a  single  number 
our  assessment  of  p^  ,  prior  to  testing  at  time  t  =  1  . 
Furthermore,  p^  is  also  to  be  used  for  determining  the  pair 
n^  and  x*  ,  for  testing  at  time  t  =  1  . 

2.  We  thus  replace  p^  by  p^  in  (2)  and  (3),  and  simultane¬ 
ously  solve  these  to  obtain  and  x*  .  [In  Appendix  A 

we  discuss  how  to  obtain  and  x*  without  using  p^  , 

and  by  directly  solving  (4)  and  (5).] 

3.  We  take  a  sample  of  size  and  test  these  to  determine  x^  , 

the  number  of  missiles  that  fire  successfully.  If  x^  >(<)  x*  , 
we  accept  (reject)  the  hypothesis  that  the  reliability  of  the 
missile  arsenal  at  time  1  is  Pj  • 

4.  If  we  accept  the  above  hypothesis,  then  we  update  our  opinions 
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about  in  light  of  and  x^  via  the  posterior 


distr ibuti 


ion  g ( p ^  |  (n^,x^),  H)  .  The  .-nodal  value  of  this 


posterior  distribution  is 


Y+xx-l 

P1  "  Y+6+n1-2  ’ 


and  this  number  best  summarizes  our  assessment  of  p^  after 
testing  at  time  1.  We  now  go  to  step  5. 

5.  If  the  aforementioned  hypothesis  is  rejected,  our  choice  of 

Y  and  6  needs  to  be  revised.  This  should  be  done  follow¬ 
ing  a  more  detailed  analysis  about  p  .  We  then  go  back  to 

X  1 

stage  1. 

6.  The  posterior  distribution  g(p^  |  (n^.x^),  H )  now  serves  as 
the  prior  distribution  for  p^  ,  and  its  modal  value  p^  is 
set  equal  to  p2  .  Thus 

Y+x  -1 

P2  =  Y+$+n  -2  » 

and  pt  is  now  replaced  by  p2  in  (2)  and  (3),  which  are 
solved  for  n2  and  x*  .  [In  Appendix  A  we  discuss  how  to 

obtain  n2  and  x^  by  directly  solving  (4)  and  (5).] 

7.  We  now  repeat  the  steps  3  through  6,  and  continue  the  above 
procedure.  Thus,  at  time  (t-1)  we  have 

Y  +  +  x2  +  ...  +  xt-l 

Pt-1  ~  Pt  '  Y  +  <5  +  nx  +  n2  +  ...  +  (7‘ 


as  our  single  best  assessment  of  the  reliability  of  the  arse¬ 
nal  at  time  (t-1),  after  observing  the  results  of  the  test  at 
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tine  (t-1)  .  It  also  represents  our  choice  for  p  in 
equations  (2)  and  (3),  for  determining  the  sample  size  n 
and  the  decision  variable  x*  . 

8.  Suppose  that  at  time  t  ,  we  test  n  items,  observe 

successes,  and  based  on  this  result,  reject  the  null  hypothe¬ 
sis  that  p  =  p  =  p  ,  .  Then  we  conclude  that  the  reli¬ 
ct  *t  t-1 

ability  of  the  arsenal  has  changed  from  its  previous  value 
Pt_^  .  When  this  happens,  we  investigate  the  cause  for  this 
change,  choose  some  new  values,  say  y'  and  6'  ,  and 
estimate  p  by 

Y'+xt-l 

^t  =  V  +  ' 

We  now  continue  as  before,  bearing  in  mind  that  the  previous 
date  (n^,x^),  ...,  are  no  m°re  appropriate  for 

inclusion  in  our  assessment  process. 

An  alternative  to  the  beta  prior  which  has  properties  of  robustness 
is  currently  under  Investigation.  However,  there  is  no  assurance  that 
the  alternative  prior  will  be  void  of  computational  difficulties. 


3. 2  Sequential  Sampling  to  Reduce  the  Amount  of  Testing 

At  any  stage  t  ,  given  an  n  and  x*  ,  a  further  reduction 
in  the  amount  of  missiles  tested  can  be  achieved  if  the  testing  is  done 
sequentially,  one  item  at  a  time.  Specifically,  we  would  test  one  item 
3t  a  time,'  and  stop  the  test  as  soon  as  the  number  of  successes  is 

larger  than  x*  .  Thus,  ideally,  the  number  of  missiles  tested  could  be 
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as  f ew  as  x*  +  1  ;  this  implies  a  saving  of  n  -  x* 


1  .  The  maxi- 


mu  it.  of  missiles  tested  would  of  course  be  no  greater  than  n^_  .  The 
resulting  sample  size,  that  is  the  number  of  missiles  actually  tested 
at  each  stage  is  known  as  a  curtailed  sample. 

For  the  above  scheme,  given  p  we  can  compute  E { n ^ ! P t )  the 
expected  number  of  missiles  tested  using  standard  arguments — these  are 
shown  later.  However,  since  p  is  not  known,  we  average  out  p^ 
with  respect  to  its  prior  distribution  to  obtain  E(n^)  ,  the  un¬ 
conditional  expectation  of  the  number  of  missiles  tested  at  each  stage 
under  the  sequentially  taken  curtailed  sample.  This  is  shown  below. 

Given  n^  and  x*  ,  the  probability  that  n  *  x  ,  when  a 
sequential  sampling  scheme  is  used  is 


VT1 


n  -x*  x-(n  -x*) 

a-Pt)  pt  .  nt 


X*  <  X  <  X* 


p{nt=x|pt}  = 


nt-x*-!  (l-pt) 


‘t~*t  x-(n  -x*) 
pt  t  t 


x-x*-l  x*+l 

t-x*-l  (1-Pt)  Pt  »  <  x  £  nt 


In  order  to  obtain  P{nt=x}  ,  we  average  out  the  above  by  g(ptl’)  , 


where 


otn  |.n  _  r(Y-HS)  Y-l,.  A 

g(Ptl  )  -  f(v)p(5)  Pt 
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When  the  above  is  done,  we  have 


r 


+ 


v 


'  X_1  1 

r(Y +£) 

F(x-n  +x*+Y)F(n  -x*+5) 
t  t  t  t 

n  -x*-l 
t  t 

r(Y)F(6) 

T(y+5+x) 

for  n 

X-l  1 

T(y-hS) 

r (x-nt+x*+Y>  (n^-x*+6) 

V*J-1 

r(Y)T(6) 

r (Y+fi+x) 

x-1 

r(Y-hS) 

(x*+1+y) T (x-x*c-l+6) 

x-x*-l 

t 

r(Y)T(6) 

r (y+£+x) 

for  x*  <  x  <  n  , 


from  which  E(n  )  can  be  computed.  The  above  formula  can  also  be  used 
to  plot  a  histogram  of  the  various  values  of  n^  ,  for  each  stage  t  . 

If  the  sequential  tests  are  to  be  done  in  batches  of  3  rather 
than  testing  a  single  item  at  a  time,  the  savings  in  the  number  of  items 
tested  will  be  less.  However,  this  is  still  better  than  compulsarily 
testing  all  the  n^  items.  We  do  not  have  a  general  formula  like  (9) 
above  to  compute  the  expected  sample  size.  The  calculations  will  have 
to  be  done  on  an  enumerative  basis.  These  are  shown  in  Appendix  C. 
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4.  COMMENTS  ON  THE  PROPOSED  APPROACH 

The  proposed  approach  is  a  combination  of  sarple  theory  and 
Bayesian  statistics.  The  former  is  used  to  determine  the  sample  size, 
and  the  latter  is  used  for  inference  about  p  .  One  may  express  reser¬ 
vations  about  a  procedure  in  which  two  philosophical  viewpoints  are  used 
simultaneously .  However,  upon  closer  examination  of  the  approach,  such 
a  concern  should  be  dispelled,  since  the  sample  theory  approach  is  not 
used  for  making  inferences  about  p^.  ;  it  is  used  for  choosing  a  sample 
size.  The  selection  of  the  sample  size  after  averaging  out  pt  with 
respect  to  its  distribution  g(pt  |  •)  ,  see  equations  (4)  and  (5),  makes 
our  analysis  fall  under  the  category1  of  what  is  knowna  as  pre-posterior 
analysis,  a  perfectly  legitimate  device  within  the  Bayesian  paradigm 
[cf.  Box  (1962)]. 

The  monitoring  of  p^_  is  done  within  the  Bayesian  framework, 
and  besides  "coherence"  it  has  the  advantage  of  inducing  economy  by 
virtue  of  the  fact  that  all  our  relevant  previous  data  are  incorporated 
into  the  analysis.  Furthermore,  it  allows  the  incorporation  of  any 
engineering  or  judgmental  knowledge  that  we  may  have  about  the  missiles 
into  our  analysis  —  this  is  done  via  the  parameters  Y  and  6  or 
Y*  and  6’  ,  etc. 

5.  APPLICATIONS  TO  DATA 

Our  proposed  approach  is  designed  to  specify  a  sample  size  for 
testing  at  each  stage,  and  thus  its  effectiveness  cannot  be  fully  ap¬ 
preciated  if  we  apply  it  to  existing  data.  However,  we  shall  apply  it 
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to  so?c  given  (sanitized)  success  failure  data  to  demonstrate  the  fact 
that  the  computations  of  Appendix  A  can  be  undertaken,  and  to  compare 
the  results  of  our  main  procedure  and  the  simplified  alternative,  de¬ 
scribed  in  Section  3.1.  In  Table  1,  we  present  the  given  success  fail¬ 
ure  data,  our  Bayesian  estimate  of  the  mode  of  pt  at  each  stage  using 
a  uniform  prior  distribution  at  stage  0  updated  at  successive  stages 
using  failure  data,  and  the  values  of  x*  and  using  the  main  pro¬ 

cedure  and  the  alternative. 

A  few  facts  emerge  from  an  examination  of  Table  1. 

1.  A  large  number  of  items  to  be  tested  is  called  for,  when 

% 

the  prior  is  uniform,  with  mode  .5  . 

2.  The  number  of  items  to  be  tested  is  the  smallest  when  the 
mode  of  p^  is  closest  to  1,  namely,  at  .9  • 

3.  The  number  of  items  to  be  tested  under  the  main  procedure 
is  always  equal  to  or  larger  than  that  under  the  alternate 
procedure.  This  is  because  the  alternate  procedure  puts  all 
the  probability  mass  at  the  mode,  whereas  the  main  procedure 
disperses  the  probability  mass  over  [0,1]  ,  with  a  concen¬ 
tration  at  the  mode. 

5 . I  Results  of  Curtailed  Sequential  Sampling 

The  sequential  sampling  approach  discussed  in  Section  3.2  was 
applied  to  the  data  and  the  results  of  Table  1.  The  nt  and  the  x* 
values  considered  were  those  given  by  the  "alternative  procedure";  this 
procedure  gave  us  smaller  values  of  the  nt's  than  the  main  procedure. 
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TABLE  1 

Results  for  Main  Procedure  and  Alternative,  Using  Sanitized 
Data,  and  Assuming  a  Uniform  Prior  at  Stage  0 


Stage 

t 

Data 

Mode 
of  P„  . 

Computed  Values  of  x* 

and  n 

t 

Main  Procedure 

Alt.  Procedure 

Success 

l 

Failure 

t 

Xt 

n 

nt 

0 

.500 

i 

2 

29 

5 

17 

1 

6 

0 

.875 

8 

13 

9 

13 

2 

11 

1 

.900  , 

10 

14 

8 

11 

3 

11 

1 

.906 

11 

15 

8 

11 

4 

11 

1 

.909 

8 

11 

8 

11 

5 

9 

3 

.875 

9 

13 

9 

13 

6 

9 

3 

.853 

10 

15 

8 

12 

7 

8 

4 

.825 

9 

14 

9 

14 

8 

4 

0 

.833 

11 

17 

9 

14 

9 

3 

2 

.820 

10 

16 

9 

14 

10 

9 

0 

.837 

10 

15 

9 

14 

11 

8 

1 

.841 

10 

15 

10 

15 

12 

7 

2 

.836 

10 

15 

9 

14 

13 

9 

0 

.848 

10 

15 

8 

12 

14 

7 

1 

.850 

10 

15 

8 

12 
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The  expected  sa~ple  sizes  when  testing  is  sequential,  in  batches 
of  3  as  well  as  one  item  at  a  time,  were  computed.  These  are  shown 
in  Table  2.  The  advantage  of  testing  one  item  at  a  time  is  clear 
from  an  inspection  of  columns  2  and  3  of  Table  2. 

We  also  note  the  overall  reduction  in  sample  size  using  the  approach 
of  this  paper.  The  expected  sample  size  can  be  as  small  as  9. 

The  detailed  calculations  leading  us  to  Columns  2  and  3  of 
Table  2  are  given  in  Appendix  C. 

6.  PROPOSED  FUTURE  WORK 

An  objectionable  feature  of'  the  proposed  procedure,  from  a 
Bayesian  point  of  view,  is  the  testing  of  hypotheses  about  p 
using  the  decision  variables  x*  ,  t  =  1,2,...  .  The  proper  Bayesian 
way  to  study  this  problem  would  be  via  a  Kalman  filter  model  which 
contains  two  unknown  states  of  nature,  p^  and  m^  ,  where  m^  denotes 
the  ^rif t  in  p^  .  The  Kalman  filter  would  not  only  have  the  ability 
to  monitor  the  reliability  of  the  arsenal,  but  would  also  provide  us 
with  a  vehicle  for  predicting  the  future  arsenal  reliability.  The 
following  are  our  ideas  on  how  a  Kalman  filter  model  for  this  problem 
can  be  developed. 


Let  Y^  denote  some  transform  of  xt^nt  »  ar>d  one  which  makes 
Y  approximately  normal.  The  observation  equation  for  the  Kalman 
filter  model  would  be 


Yt  “  Pt+Ylt 

2 

where  Y^t  is  a  disturbance  term  with  mean  0  and  variance  0^  . 
We  can  postulate  the  following  as  system  equations: 

Pt  ■  "t  +  V2t  • 


m 


"t-l  +  Y3t  ' 


and 
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TABLE  2 

Expected  Sample  Size  for  Curtailed  Sequential  Sampling  in  Batches  of 

Size  3  and  Size  1. 


Stage  1 
t 

t 

Expected  Sample  Size 
for  Batch  Size  3 

Expected  Sample  Size 
for  Batch  Size  1 

B 

nt 

0 

11.84 

10.91 

5 

17 

1 

12.03 

10.66 

|  9 

13 

2 

10.29 

9.45 

! 

1  8  ; 

11 

3 

10.37 

9.51 

8 

11 

4 

10.40 

,  9.54 

8 

11 

5 

12.28 

11.08 

9 

13 

6 

11.07 

10.16 

8 

12 

7 

12.84 

11.74 

9 

14 

8 

12.79 

11.69 

9 

14 

9 

12.87 

11.78 

9 

14 

10 

12,78 

11.67 

9 

14 

11 

13.59 

12.72 

15 

12 

12.78 

11.68 

9 

14 

13 

11.14 

10.22 

8 

12 

14 

11.14 

10.21 

8 

12 
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In  the  above  equations,  we  are  saying  that  p  ,  the  unknown  state  of 
nature,  consists  of  a  low  frequency  drift  term  m  ,  which  represents 
a  smooth  variation  in  ,  and  ^£t  *  which  is  a  high  frequency  compo¬ 

nent  that  represents  drastic  changes  in  pc  ,  We  assume  that  y^  is 

2 

a  normal  variate  with  mean  0  and  variance  .  The  drift  term  is 

assumed  constant,  except  for  slight  disturbances  in  it;  these  are 

described  by  >  which  is  also  assumed  normal  with  mean  0  and  vari- 

2 

ance  . 

The  Kalman  filter  solution  would  result  in  uncertainty  statements 
about  pt  and  m  ,  via  their  distribution  functions.  These,  of  course, 
would  be  conditioned  on  (n^x^,  . .  .  (nt,xt)  .  Large  values  of  m 

would  indicate  a  drift  in  the  arsenal  reliability,  and  so  m  could  be 
used  to  monitor  the  change  in  the  arsenal  reliability. 

It  appears  that  the  Kalman  filter  solution  would  have  several 
advantages  over  the  proposed  approach.  The  problem  of  choosing  n^  in 
the  context  of  a  Kalman  filter  is  an  open  question,  and  this  calls  for 
some  basic  research,  assuming  that  this  has  not  been  done  before. 

A  third  possible  direction  for  future  research  is  the  development 
of  a  sequential  procedure  for  testing  the  missiles.  A  sequential  proce¬ 
dure  employing  Bayesian  considerations  may  add  a  further  dimension  to 


this  problem. 


Chapter  IV 

Woodroofe's  Proposal 


The  proposals  of  Michael  Woodroofe  are  not  yet  formally 
documented,  but  are  contained  in  a  series  of  letters  and  lecture 
notes  (References  13-17).  In  this  chapter  I  shall  mostly  quote  from 
this  material  with  the  author’s  permission,  noting  that  any  published 
versions  may  differ  markedly  from  those  given  here.  I  accept 
responsibility,  however,  for  the  accuracy  of  the  material  quoted  and 
the  interpretations  and  extensions  of  it. 

All  of  the  calculations  described  in  this  chapter  were  carried 
out  by  Dr.  Woodroofe  and/or  myself.  I  have  programmed  most  of  them 
for  an  HP-41C,  and  listings  are  given  in  Appendix  D.  Instructions 
and  copies  on  magnetic  cards  are  available.  Dr.  Woodroofe  has  used 
an  Apple  computer. 


Section  1.  (Extract  from  Reference  14). 

The  Truncated  Sequential  Probability  Ratio  Test. 

Illustration  with  a  sequential  test  of  the  type  of  savings 
which  are  possible  and  the  loss  of  information  which  results  from  the 
savings.  Note  that  the  process  starts  with  the  conventional 
Uniformly  Most  Powerful  test,  to  be  terminated  when  a  specific  number 
Sn  of  failures  has  been  observed;  or  when,  out  of  a  planned  test  of 
size  n,  the  number  of  observed  successes  assures  that  the  number  of 
failures  cannot  reach  Sn ;  or  after  n  tests  if  not  terminated  earlier. 
The  choice  of  n  is  at  this  time  arbitrary;  the  value  12  was  used  in 
the  example  to  permit  comparison  to  the  Pershing  test  program,  past 
and  planned. 


/We  start  with  a  discussion  oU  the  problem  of  sequentially  testing 
/such” that7  that  a  failure  probability  does  not  exceed  a  given  level.  I  will 
illustrate^  the  type  of  savings  which  are  possible  and  the  loss  of  infor¬ 
mation  which  result  from  the  savings  with  a  specific  example. 

Let  X.  ,  ...  X..  be  1. 1. d.  random  variables  which  take  the  values 
1  and  0  witn  probabilities  p  and  q  *  1-p  ,  where  0  <  p  <  1  ,  is 
unknown;  and  consider  the  problem  of  testing 


H  :  P  <  -15  . 


Let  S  -  X.  +  ...  +  X.  , 


1  <  k  <  12  . 


Tv 

Then  the  (UMP)  test  which  rejects  H  If  and  only  if  S  '>  4  has  power 
function  ° 

(1)  -B  (p)  -  1  -  I  f pk  q"'k  ,  0  <  p  <  1  . 

°  k-0 


( 1 2  \  k  n-k 

Ur  q  • 


Of  course.  It  may  not  be  necessary  to  take  all  12  observations  to  determine 
whether  S(2  >  ^  .  The  test  may  be  curtailed  at  time 

tQ  ■  mln{k  1  :  4  or  <_  k-9)  . 


(2)  E  (t  )  -  l 

p  °  k-4 


■  <■') 


4  k-4 

P  q 


i  <v) 


9  nR"9 

q  p 


o  <  p  <  1 


*  Identically  and  Independently  Distributed. 
**  Uniformly  Most  Powerful. 


is  the  expected  sample  size  of  the  curtailed  test. 

Selected  values  of  &D(p)  and  Ep(tQ)  are  ^stec*  columns  2 
and  4  of  Table  I  below. 

Observe  that  the  type  I  error  probability  Is  .0922  when  p  »  .15  and 
the  type  II  error  probability  Is  .2253  when  p  ■  .4. 

I  tried  to  construct  a  truncated  version  of  the  SPRT  whose  power 
function  matched  as  closely  as  possible.  Wald's  approximations 

allow  one  to  match  the  power  function  at  two  points.  I  picked  .15  and 
.40.  Wald's  approximations  then  give  formulas  for  the  upper  and  lower 
stopping  boundaries  in  the  (k,  S.)  plane.  These  are  listed  in  columns 
2  and  3  of  Table  2.  There  are  two  problems  with  these  boundaries:  Wald's 
approximations  tend  to  overestimate  the  error  probabilities;  and  I  wanted 
the  test  to  take  at  most  )2  observations.  After  some  experimentation 
with  formulas  (3)  and  (4)  below,  I  was  led  to  the  upper  and  lower 
boundaries  listed  in  columns  4  and  5  of  Table  2. 

Thus,  I  considered  the  sequential  test  which  takes 

t  •  min{k  1 :  a^  or  b^) 

observations  and  rejects  H  if  and  only  if  S  >  b  ,  where  a,  and 

b.  are  as  in  Table  2. 
k 

The  power  function  and  expected  sample  size  may  be  easily  computed. 


fk(J’p)  "  Pp{Sk"J  »  1  >  k) 

for  k  «  0  ,  ...  ,  11  ,  j  ■  0,  1,2  ,  ...  ,  and  0  <  p  x  1  .  Then  the 
power  function  and  expected  sample  size  are 

(?)  Btp)  •  Vi'V-  p)  '  p  - 

and 

12 

(3  Ep(t)  ‘  kE,  k(,k-i(bk'1,  p)  p  *  fk-i(V  p)<,) 

for  0  <  p  <  1  .  Thus,  one  need  only  compute  the  values  of  f.  ;  and  this 
is  easy  in  view  of  the  initial  conditions,  f  (0,p)  ■  1  and 
fg(Jip)  •  0  for  J  i<  0  ,  and  the  recursion  ® 

(5)  fk(J,p)  "  lp  fk-l^’,,p^  +  q  fk-l^»p^  '*ak  <  J  <  bk^ 


V 


\for  k  «  I  ,  ...  ,  12  ,  J  ■  0,  1,2,  ...  ,  and  0  <  p  <  1  .  Here  1^ 
denotes  the  Indicator  of  A. 

The  power  function  and  expected  sample  size  may  be  computed  from 
(3)>  (4),  and  (5).  Selected  values  are  listed  in  columns  3  and  5  of 
Table  1. 

Observe  that  the  power  functions  3  and  3  differ  by  at  most 
.0103  for  the  values  computed.  This  is  much  better  than  I  had 
expected  when  I  began  the  exercise.  Observe  also  that  the  expected  I 
sample  size  of  the  modified  SPRT  is  substantially  smaller  than  that  of  | 
the  curtailed  test  when  p  Is  small. 

After  the  test  has  been  performed,  one  may  set  confidence  limits 
for  p  by  using  the  relationship  between  tests  and  confidence  Intervals. 
Order  the  possible  outcomes  In  a  clockwise  manner,  as  in  column  1 
of  Table  3.  For  each  r  ,  0  <  r  <  1  ,  one  may  test  the  hypothesis 

V  P  -  r 

as  follows:  the  acceptance  region  A(r)  of  the  test  consists  of  an 
Initial  segment  of  outcomes.  In  the  order  of  Table  3;  one  includes 
precisely  enough  outcomes  to  make 

Pr(A(r)  )  >.  .90  . 

* 

Then,  after  the  test  has  been  performed,  an  upper  confidence  bound  p 
for  p  may  be  obtained  from  the  relation 

i 

p  <_  p*  iff  (t,St)  e  A(p)  . 

This  is  essentially  the  approach  of  Siegmund  (1978,  Biometrlka),  but 
substitutes  exact  calculations  for  his  approximations. 

I  list  some  approximate  75%  upper  confidence  bounds  for  p  in  Table  3 
These  were  obtained  by  linear  interpolation  with  formulas  like  (3). 

i 

To  the  extent  that  the  modified  sequential  test  takes  fewer  observe- 
tions  than  the  curtailed  test,  one  may  expect  less  accurate  estimation  of 
P* 


p 


.05 

.10 

.15 

.20 

.25 

.30 

.1*0 

.50 


Table 


1 ;  Power  Functions  and  Expected  Sample  Sizes 


(Up) 

.0022 

.0022 

.0256 

.0251 

.0922 

.0899 

.205** 

* 

N) 

O 

o 

-E“ 

.3512 

.  3I<31* 

.5075 

.1*975 

.771*7 

.76^ 

.9270 

.9201* 

E  (t  ) 

E  (t) 

p  o 

p 

9  - 

6.93 

9.92 

7.85 

10.23 

8.62 

10.1*0 

9.13 

10.31 

9.35 

10.02 

9.30 

9.00 

8.57 

7.77 

7.»*2 

Here:  Column  1  Is  computed  from  (l),  column  2  from  (3),  column  3 
from  (2),  and  column  k  from  k. 


Table 

2:  Upper  and  Lower  Stopping 

Boundaries  In  the 

(k-  y 

P 1  ane 

The  SPRT 

Modi  fled 

k 

* 

\ 

* 

bk 

ak 

bk 

1 

-1 

2 

>1 

3 

2 

-1 

3 

-1 

3 

3 

-1 

3 

-1  • 

3 

k 

-l 

3 

-1 

l* 

5 

0 

3 

-l 

i* 

6 

0 

1* 

0 

i* 

7 

0 

i< 

0 

i* 

8 

1 

i* 

0 

k 

9 

1 

1* 

l 

k 

10 

1 

5 

I 

1* 

11 

1 

5 

2 

l* 

12 

2 

5 

3 

i* 

Here  columns  2  and  3  are 
are  ad  hoc  approximations 

f  rom  Val d 

's  approximations; 

columns  1* 

and  5 

—  Approximate  75*  Upper  Confidence 


Outcome 

t  S 

t 

3  3 

5  4 

6  4 

7  4 

8  4 

9  4 

10  4 

11  4 

12  4 

12  3 
11  2 
9  1 

6  o 


Confidence  Bound 


.91 
-70 
.61 
.55 
.  5 


.45 
.42 
-39 
•  34 


.29 


.21 


Bounds 


('orient-  by  DW: 

As  indicated  in  Chapter  III,  expectations  of  ’  and  E  can  be 
computed  based  on  a  prior  probability  distribution.  Closed-form 
solutions  exist  for  j;  o  and  Ep(to)  for  a  Beta  prior,  among  others. 
For  ’•  ( p )  ,  and  Ep(t),  numerical  integration  is  necessary.  Other 
indices  derived  from  the  fk(j,p)  in  manners  like  that  for  p  or  E(t) 
can  also  be  meaningfully  be  averaged  over  a  prior  distribution.  As 
Tp(t)  has  here  a  narrow  range  of  variation,  its  expectation  value 
vill  not  be  very  sensitive  to  the  choice  of  the  prior  distribution. 


Section  2 .  (Extract  from  Reference  15). 


To  clarify  some  of  the  points  raised  in  Section  I,  V’oodroofe 
provided  a  more  extensive  treatment  of  the  development  of  the  limits 
on  observed  successes  and  failures  at  which  the  test  is  terminated. 
It  begins  with  the  method  described  by  Wald  (op.  cit.)  and  then 
continues  with  a  procedure,  somewhat  judgmental,  for  modifying  those 
boundaries  to  reduce  the  expected  size  of  the  test  while  retaining 
itspower. 


1)  Testing  Hq:  6  >  .15  is  the  same  as  testing  6'  =1-6  <  .85.  If 
you  want  to  have 


Pe{decide  6'  >  .85}  <  oq  for  6'  <  .85 


and 


Pg{ decide  6*  <  .85}  <  cxi  for  all  6*  >  6'  >  .85, 

1 


where  oq  ar*3  are  snail  and  .85  <  6^  <  1,  then  you  cannot  simply  reverse  the 

roles  of  zero  and  1  in  the  test  described  in  my  earlier  letter.  A  new  test 
must  be  constructed.  See  (2)  below. 


In  /Section_I7  0  was  the  probability  of  a  system  failure. 


2)  For  testing  Hq:  6  <  60  at  level  oq  with  type  II  error  at  nost  a* 
when  6  >  ejf  where  0  <  6g  <  6*  <  1  are  specified,  the  Sprt  continues  sampling 
as  long  as 


1/A  <  Ln  <  B 


where  B  =  (l-ai)/a0,  A  =  (l-ag^n,  and  L„  is  the  likelihood  ratio.  One  finds 


-2- 


Ln  =  exp  {A2Sn  -  n  A0) 

where 

Al  =  leg  82(1-0q)  "  leg  eo^l_el^ 


&q  =  log  (1-8q)  -  log  ( 1 — ® 1 ) 

and  Sn  =  X1+...+Xn,  n  >  1. 

Since  Sn  are  integer  valued,  equation  (*)  may  be  rewritten 

an  *  Sn  <  bn 

<U'  ^ 

an  *  1  Ta(nA0  *  lo9  A)1 

cu  _ 

bn  *  l  j  (nAQ  +  log  B))  +  1 

where  [x]  is  the  greatest  integer  which  is  less  than  or.  equal  to  x. 

Suppose  now  that  one  wants  the  test  to  be  truncated  at  M  say.  Then  one 
wants  boundaries  an  and  b^,  1  <  n  <  M.  What  I  did  in  the  example  was  the 
following.  Let  and  b^  be  such  that 

aM  <  a.  =  by  r  1  and  , 

n  n  n  n  n 

say  two  integers  near  the  middle  of  the  interval  from  to 

+ 

an  =  max  (an,  a^  (M  -  n)  \ 

and  b  =  min  (b  ,  bM) 

n  n  n 


'  fo^r  n  <  M.  This  gives  a  first  approximation  to  the  boundary.  In  the  example, 
I  then  oerputed  the  power  function  of  the  sequential  test  with  boundaries  an 
and  ty,  and  compared  it  with  the  power  function  of  the  fixed  sample  size  test. 

I  then  changed  a  few  of  the  boundary  points  to  get  better  agreement  between 
the  two  power  functions.  The  adjustments  were  minor  and  tended  to  make  the 
continuation  region  fatter. 

The  reason  that  you  can’t  pin  me  down  on  the  adjustments  is  that  it  is  a 
triad,  and  error  operation. 


h^.  Then  let 
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(3)  In  the  example, 


Pe{t=k,Sk  =  bfc}  =  fk-l^  “  1  6 ) *  S 


and 


PgttAjSfc  =  a^}  =  f)c-lfak'e)*  (3_B) 


Then  Pe{Xt  >  x}  is  the  sum  of  these  probabilities  over  all  pairs  (k^a^)  and 
(k,^)  for  which  a^/k  >  x  or  bj</k  >  x. 


4)  For  inverse  sampling  there  is  just  one  boundary.  Ftor  curtailed 
sampling,  there  are  two.  Let 

t+  =  minlk  >1:  >  4) 


and 


t~  ;=.  min{k  >1:  k  -  >  9) 


Then  Eg(t+)  -  4/0 

and  E8(t")  =  9/ (1-8) 

The  stopping  time  for  the  curtailed  fixed  sample  size  test  is 

to  =  ndn(t+,f-) 

I 

*  So  E6(to)  <  min{E0<t+)f*Ee(t-)} 

i 

When  6  =  .15,  Ee(t~)  *=  10.6. 

The  formulas  for  Ee(t+)  and  Egft")  hold  for  all  0  <  6  <  1. 

* 

5)  I  think  of  the  boundaries  as  a  modified  S.P.R.T.  In  the 
example,  they  were  similar  to  the  curtailed  fixed  sample  size  test,  but 
sufficiently  different  to  reduce  the  expected  sample  size  by  about  1  over  the 
range  of  interest. 

The  calculations  in  ny  letter  to  Launer  are  for  fixed  6.  To  do 
.  a  Bayesian  calculation,  one  would  average  them  over  8  values 


The  formulas  which  I  gave  for  computing  the  power  and  expected  sample 
implicitly  assume  that  that  the  bo.  r.J.  ries  an  and  bn  are  non-decreasing  in  n. 


Section  3  (Extract  from  Reference  16). 

The  Truncated  SPRT,  Aggregated  over  Several  Tests. 

Derivation  of  a  conservative  estimate  of  the  probability 
that  in  10  years  of  testing,  at  12  missiles  planned  for  expenditure 
each  year,  no  more  than,  say  100,  will  be  needed  using  the  proposed 
stopping  rules. 


This  is  to  explain  how  savings  in  expected  sample  size  may  be 
translated  into  savings  of  units  which  must  be  purchased  prior  to  the 

experimentation.  For  definiteness,  I  illustrate  the  method  with  the  - 

truncated  SPOT,  which  is  described  in  /  Section  1/  ^ 

In  particular,  recall  the  computation  of 

f(k,j;p)  =  PR(T>k,Sk=j), 

where  p  denotes  the  true  failure  probability,  Sk  denotes  the  number  of 
failures  after  k  units  have  been  tested,  and  t  denotes  the  stopping  time. 

Fran  this,  one  gets  V 

'  k 

G(k;p)  *  Pr(T<k)  «  1  -  Ij«0 
and  g(k;p)  =  Pr(T=k)  =  G(k;p)  -  G(k-l;p) 
for  k  *  1,...,12  and  0  <  p  <  1. 

Suppose  that  the  truncated  SPOT  is  run  n  times,  say  once  each  year 
for  n  years,  where  n  is  a  positive  integer.  Then  there  will  be  a  -sequence 
Pl»***'Ph  unobservable  true  failure  probabilities  and  a  sequence  t^,...,^ 
of  random  sample  sizes.  Here  I  regard  Pjy*»Pn  as  unknown  parameters,  and 
suppose  that  t^,...#^  are  independent  randan  variables  for  which 

Pr(tj=k)  =  gfkjp^ 

for  k  *  1,...,12  and  i  =  1,.. .  ,n.  If  Pi»**-»Pn  are  really  randan  variables, 
then  the  calculations  described  below  are  valid,  if  the  conditional  distri¬ 
bution  of  tj,...,^  given  pi# •«.»Pn  is  as  just  described. 
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Let  T  denote  tbc  total  number  of  units  used  during  the  tests, 

T  =  tj+^.+V 

Then  the  distribution  of  T  is  required.  The  distribution  of  T  is  the 
convolution  of  the  individual  distributions  of  ti,...,^.  This  depends 
on  Pi/-*»/Pn  in  a  complicated  manner,  but  it  is  possible  to  find  the  sharp 
bound  which  is  valid  for  all  possible  choices  of  Pi,...,pn.  That  is,  it  is 
possible  to  find  a  function  H  for  which 

Pr(TOc)  >  H(k) 

for  all  k  *  l,...,12n  and  all  possible  choices  of  plf... ^ 

I  describe  the  derivation  be  lew, 

The  values  of  h]  are  included  in  Table  2  in  the  special  case  that 

~n  *  10.  Observe  that  then 

Pr(T  >  105)  <  .054 

for  all  pi,.*.#Pn.  The  bou*^  is  reasonably  sharp,  since  Pr(T>105)  *  .050 
when  all  of  Pi » •  •  •  »Pn  ore  e<3ual  *27. 

While  the  bound  is  sharp,  the  approach  is  conservative,  since  it 
ignores  data  from  previous  years  and  assumes  the  worst  possible  values  for 
Pl»*  *  *  ,Pn*  If  30  indePendent  verification  is  required  for  each  year,  then 
seme  of  this  conservatism  may  be  unavoidable. 

The  derivation  of  the  bound  uses  the  notion  of  stochastic  dominance. 

If  X  and  Y  are  random  variables  with  distribution  functions  F  and  G,  then  Y 
is  said  to  be  stochastically  larger  than  X  if  and  only  if  G(z)  <  F(z)  for 
all  z.  If  X  and  X'  are  independent  random  variables  and  Y  and  Y'  are 
independent  random  variables  and  if  Y  and  Y'  are  individually  stochastically 
larger ^dX  and  X',  then  Y+Y'  is  stochastically  larger  than  X+X'  (as  is 
easily  verified);  and  this  result  extends  from  two  summands  to  several. 

To  apply  this  result,  let 

G(k)  *  min  G(k;p), 

where  the  minimum  extends  over  0  <  p  <  1.  Then,  for  any  choice  of  p^,...fp_, 
the  distribution  of  T  is  stochastically  dominated  by  the  sum  of  n  independent 
random  variables  having  common  distribution  function  G.  Computing  G  is 
straightforward.  For  k  <  6,  the  minimum  is  attained  when  p  =  0  and  G(k)  *  0. 
For  k  >  6,  I  confuted  G(k;p)  for  a  grid  of  p  values  and  took  the  minimum  over 
this  grid.  The  values  nre  listed  in  Table  1.  I  used  a  grid  width  of  .01. 
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TABLE  1.  Values  of  G(k;p) 


p 

k  =  6 

7 

8 

9 

10 

11 

.24 

.2313 

.2590 

.2966 

.5032 

.5556 

.7723 

.25 

.2222 

.2535 

:  2955 

.4967 

i  5537 

.7685 

.26 

.2144 

.2496 

.2962 

.4923 

.5538 

.7661 

.27 

.2081 

.2474 

.2987 

.4900 

.5559 

:7651 

.28 

.2032 

12468 

.3030 

14897 

.5599 

.7654 

.29 

.1996 

.2477 

.3088 

.4914 

.5657 

.7669 

.30 

.1974 

.2501 

.3163 

.4949 

.5731 

.7696 

.31 

:i?§4 

.2540 

.3252 

.5002 

.5819 

.7735 

.32 

.1967 

.2593 

.3355 

.5072 

.5922 

.7783 

.33 

.1981 

.2659 

.3472 

.5156 

.6036 

.7840 

.34 

.35 

<V- 

b 

C- 

A 

<3. 

X 

Minimum 

.1964 

.2468 

t 

.2955 

.4897 

.5537 

.7651 

< 

Mean  and  St  dev  y  =  9.4528  "  o  =  2.1992 

-  c  .  -C) 

C  ~  4- 0°+9)  A-  fr-Ot-- b-  (»<.)  «L 

Notes:  G(12;p)  =  1  for  all  0  <  p  <  1;  the  minimum  is  zero  for  k  <  5;  y  and  o 
are  the  standard  deviation  of  the  minimizing  distribution. 
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TABLE  2.  Values  of  H 


1  -  H(10  H(k>  -  H(Jc-l) 


100 

.2026 

.0460 

101 

.1622 

.0404 

102 

.1273 

.0349 

103 

.0978 

.0295 

104 

.0734 

.0244 

105 

.0537 

.0197 

106 

.0382 

.0155 

107 

.0263 

.0118 

108 

.0175 

.0088 

109 

.0112 

.0063 

110 

.0069 

.0043 

111 

.0040 

.0029 

112 

.0022 

.0018 

113 

.0012 

.0011 

114 

.0006 

.0006 

115 


0002 


0003 


Comments  by  DW: 

Let  g ( k )  =  G ( k) -G ( k-1 )  . 


4.1 


Then  d (n , z  ) 


z^g(n-k) 


4.2 


is  a  generating  function  of  the  distribution  g(k).  The  generating 
function  for  the  dominant  of  m  years'  test  results  is  then 


rim 


r\ 


4.3 


J=e3 


and  the  dominant  of  the  probability  that  a  specific  number  J  of  tests 
can  be  forgone  is  given  by  the  coefficient  dJ  of  zJ  in  the  expansion 
of  D ( n , m ) . 

In  our  example  n=12,  and  the  g(k)  for  k  <  6  are  all 
zeros.  Sample  data  are  given  in  Table  3.  So,  for  m=10, 


D  = 

*  iV'1!  l O Tz  •+  "  •  . 


4.4 

L 


\o 


TABLE  3 


g(k) 


P  =  .85 
Batch  Size 


P  =  .75 
Batch  Size 


k 

1 

3 

12 

.2349 

.5103 

11 

.2114 

0 

10 

.0640 

0 

9 

.1942 

.2933 

8 

.0487 

0 

7 

.0504 

0 

6 

.1964 

.1964 

1 

3 

0940 

.4433 

1258 

0 

2235 

0 

1694 

.3604 

0361 

0 

1549 

0 

1963 

.1963 

In  Vo odroofe's  notation 

t-A  C™- 3- 1). 

In  particular,  in  our  case, 

H0'<0  --  ! - 

is  the  dominant  of  the  probability  that  all  120  are  required  (none 
can  be  foregone).  It  follows  that 

\-  AA(vim-I-t) 

£> 

is  the  dominant  of  the  probability  that  at  most  J  can  be  forgone;  the 
generating  function  for  eJ  is 

-r  .  . 

t  <2z  r  DC'^/G -*). 

d 

The  calculation  of  the  dJ  or  eJ  presents  no  difficulty  except 
possibly  in  the  control  of  round-off  errors  for  J  large.  Sample 
results  are  given  in  Tables  4  and  5  partly  repeating  material  in 
Table  2,  with  differences  presumably  due  to  differences  in  accuracy 
between  our  computers. 

In  actual  conduct  of  Follow-on  Tests,  three  failures  in  a  row, 
or  two  with  an  identifiable  cause,  would  be  sufficient  justification 
for  halting  the  test  until  the  problem  were  (identified  and)  fixed. 
There  would  then  remain  some  number  of  missiles  from  that  year's 
allocation  available  for  intensive  investigation  of  the  fault  and  for 
demonstration  of  remediation.  It  is  not  clear  that  any  additional 
missiles  would  need  to  be  allocated  to  those  missions,  as  they  could 
serve  the  FOT  mission  at  the  same  time. 

It  is  a  trivial  matter  to  revise  the  expression  for  D(n,m)  to 
treat  the  case  of  batched  tests:  for  example,  in  groups  of  3. 

Tables  3-5  compare  the  results  for  single  and  triple  tests.  For  the 
data  in  the  example,  whatever  the  number  of  missiles  considered  an 
adequate  inventory  for  10  years'  testing  without  batching,  about  6-10 
more  would  be  required  when  fired  in  batches  of  3.  The  analysis  in 
Chapter  III  gave  a  similar  result. 

Up  to  this  point  the  development  has  assumed  that  up  to  12 
would,  in  fact,  be  expended  if  necessary  to  provide  the  foundation 
for  an  annual  confidence  estimate.  The  question  now  is:  why 


TABLE  4 


P  =  .85 


Singles 


k 

d  J= 

e  J  = 

H(k)-H(k-1) 

l-H(k) 

120 

5.1E-7 

5.1E-7 

119 

4.5E-6 

5.1E-6 

118 

2.0E-5 

2.5E-5 

117 

.0001 

.0001 

116 

.  00c  1 

.0002 

115 

.0003 

.0006 

114 

.0006 

.0012 

113 

.0011 

.0022 

112 

.0018 

.0040 

111 

.0029 

.0069 

110 

.0043 

.0112 

109 

.0063 

.0175 

108 

.0088 

.0263 

107 

.0118 

.0382 

106 

.0155 

.0536 

105 

.0196 

.0733 

104 

.0243 

.0975 

103 

.0292 

.1268 

102 

.0342 

.1609 

101 

.  0392 

.  2001 

100 

.0439 

.2441 

Batches  of  3 


d  j  e  J  J 

0012  .0012  0 

1 

2 

0069  .0081  '  3 

4 

5 

0224  .0305  6 

7 

8 

0511  .0816  9 

10 

11 

0902  .1718  12 

13 

14 

1291  .3010  15 

16 

17 

1545  .4554  18 

19 

20 


TABLE  5 


Singles 

d J=  e J= 

k  H(k)-H(k-1)  l-H(k) 


120 

5E-11 

119 

7E-10 

118 

6E-9 

117 

3E-8 

116 

1  .  4E- 7 

115 

5.5E-7 

114 

2.0E-6 

113 

5.0E-6 

112 

1 .4E-5 

0 

111 

3.0E-5 

.0001 

110 

.0001 

.0001 

109 

.0002 

.0003 

108 

.0003 

.0006 

107 

.0005 

.0011 

106 

.0010 

.0021 

105 

.0016 

.0037 

104 

.0025 

.0062 

103 

.0039 

.0101 

102 

.0057 

.0159 

101 

.0082 

.0241 

100 

.0113 

.0354 

99 

.0152 

.0505 

98 

.0197 

.0702 

97 

.0248 

.0950 

96 

.0304 

.1255 

95 

.0364 

.1618 

=  .75 

Batches  of  3 


dJ  eJ  J 

.0003  .0003  0 

1 

2 

.0024  .0027  3 

4 

5 

.0100  .0127  6 

7 

8 

.0284  .0411  9 

10 

11 

.0604  .1014  12 

13 

14 

.1016  .2031  15 

16 

17 

.1401  .3432  18 

19 

20 

.1615  .5047  21 

22 

23 

,1578  24 

25 


annually?  If  an  annual  series  should  end  without  clear  resolution, 
as  indeed  it  must  occasionally  according  to  the  current  plans  what 
then?  If  there  is  not  a  clear  cause  of  alarm,  there  is  no  need  for 
alarm . 

Consider  a  decision  to  limit  the  annual  expenditure  to  9 
missiles,  while  extending  the  reporting  period  to  cover  12  missiles 
(the  current  standard)  if  uncertainty  had  not  been  earlier  resolved. 
In  the  worst  case  (all  12-missile  series)  reports  would  occur  at  16- 
month  intervals,  or  8  reports  in  11  years.  Were  the  JCS  to  accept 
biennial  reporting  as  an  (occasional)  substitute  for  annual 
reporting,  this  would  be  a  technically  simple  solution. 


Sect  ion  4  (Extract  from  Reference  17) 


A  Completely  Bayesian  Stopping  Algorithm 

[This  is  my  suggestion  for  doing  a  complete  Bayesian]  decision  theoretic 
analysis  of  the  stopping  problem.  On  the  basis  of  the  preliminary  calculations 
described  below,  I  estimate  that  this  approach  would  reduce  the  number  of  units 
needed  for  testing  by  at  least  one  per  year  over  the  savings  which  may  be 
attained  by  using  a  sequential  probability  ratio  test. 


The  approach  requires  the  specif ica tion  of  a  prior  distribution  and  a 
loss  structure.  I  suggest  a  possible  form  for  these  quantities  below;  but 
other  choices  would  yield  to  similar  analyses. 

Let  p  denote  the  proportion  of  non-defective  items  in  the  population. 

Let  h|  denote  a  density  on  the  unit  interval,  0<p  <1;  let  hg  denote  the 
uniform  density  on  the  unit  interval;  and  consider  prior  densities  of  the  form 

(1)  g(p)  =  w  h1(p)  +  (l-w)hQ(p), 

where  0  <  w  < 1  is  a  prior  parameter.  Here  h^  may  be  thought  of  as  the 
posterior  density  which  resulted  from  last  year’s  tests,  and  w  is  the 
probability  that  p  hasn't  changed  during  the  past  year.  If  p  has  changed, 
which  it  may  with  probability  1-w,  then  it  is  assumed  to  be  uniformly 
distributed  over  the  interval  0  <  p  <1. 


Suppose  now  that  one  may  observe  conditionally  independent  Bernoulli 
randon  variables  X^,...,X^  with  common  success  probability  p,  given  p,  and  let 


S 


k 


V---+Xk 


denote  the  number  of  successes.  Then  the  posterior  distribution  of  p,  given 

X. , . . . ,X  is 
i  n 

gk(p)  =  w  hk  (p)  +  (l-w)hg(p) 

.  S  k_Sk 

where  h^p)  =  h^pjk.S^)  a  p  (1— p)  ^tuCp) 

hk(p)dp=l 


and 


2 


^  Suppose  now  that  a  critical  level  pp  is  given  with  the  following 
properties:  if  p  >  Po,  then  the  population  contains  enough  good  itens;  if 
p  <  Pq,  then  the  population  no  longer  contains  enough  good  items  and 
corrective  action  is  desirable;  and  if  p  is  much  less  than  pQ,  then  corrective 
action  is  necessary.  Suppose  further  that  the  purpose  of  each  year's  test  is 
to  decide  whether  p  <  Po  or  p  >.Pq;  and  define  one  unit  of  cost  to  be  the  cost 
of  testing  one  item.  Thai  the  decision  problem  may  be  modelled  as  follows: 
the  possible  decisions  are  1  to  decided  that  p  <  pg  and  2  to  decide  that 
p  >  pq;  if  one  decides  that  p  <  pg  when,  in  fact,  p  >pq,  then  one  loses 
units;  and  if  one  decides  that  p  >  Pq,  vhen,  in  fact,  p  <  pq,  then  one  loses 
C2(po~P)  units.  Hare  and  C2  are  positive  constants.  represents  the 
cost  of  inspecting  the  entire  system;  and  the  ratio  C2/C1  is  determined  by  the 
relative  importance  of  the  two  kinds  of  errors. 

These  three  elements,  the  prior  distribution,  the  sampling  distributions, 
and  the  loss  structure,  determine  an  optimal  sampling  plan,  one  which 
minimizes  the  sum  of  sampling  costs  and  expected  loss  to  due  an  incorrect 
decision.  To  describe  it,  first  let  m  denote  the  maximum  number  of  tests 
which  could  be  conducted  in  any  given  year  (e.g.  m  =  12).  Next,  let 

Li(k,s)  =  C]P(p  >  PolS^s)  +  k 

and  L2 ( k , s )  =  C2E{nax(0,po  -  p) 1 =  s)  +  k  " 

for  k  =  0,...,m  and  possible  values  of  s.  Thus  and  L2  denote  the 
conditional  expected  losses  for  the  two  decisions,  given  X^,...^,  plus  the  . 
cost  of  observing  X2,...,X}C.  If  k  =  0,  then  s  =  0  and  the  expectations  are 
unconditional.  If  sampling  is  terminated  after  k  tests,  then  it  is  optimal  to 
make  decision  1  if  and  only  if  LitkjSfc)  <  L2(k,Sjc),  in  which  the  expected  loss 
due  to  terminal  decision  is  * 

/ 

LfltkjSfc)5*  min{Li(k,S>;)  ,L2(k,Sx). 

Let  p(k,s)  =  P(Xk+i  =  1  |  Sfc  *  s) 

for  k  =  l,...,m-l  and  possible  values  of  s;  and  define  L  by 

L(m,s)  =  I<)(m,s) 


and  L(k,s)  =  min  {Lo(k,s), 


(2)  p(k,s)L(k+l,s+l)  +  (l-p(k,s$L(k+l,s)} 

for  k  =  0,...,m~l  and  possible  values  of  s.  Then  the  optimal  sampling  plan  is 
to  continue  sampling  as  long  as  L(k,S^)  <  Lo(k,S)<),  stopping  at  time 


-3- 


t  =  min{h>0:LQ(k,S)c)  =  l/k^)}. 

Here  L(k,s)  is  the  minimum  expected  loss  plus  sampling  cost  among  all  sampling 
plans  which  take  at  least  k  observations. 

If  h  is  a  beta  density,  then  it  is  possible  to  compute  Lp  and  L2  as  sums 
of  products  of- pg  and  (1-po)  times  ratios  of  factorials.  I  can  supply  the 
details,  if  you  are  interested.  Using  these  explicit  expressions,  it  is 
straightforward  to  compute  L  by  the  backward  induction  (2);  and,  once  L  and  Lg 
have  been  computed,  it  is  simple  to  classify  the  possible  outoomes  (k,s)  as 
stopping  points,  points  for  which  Lg(k,s)  =  L(k,s),  or  continuation  points. 
Moreover,  the  stopping  points  divide  thsnselves  into  lower  stopping  points  for 
which  L<)(k,s)  =  Lp(k,s)  and  upper  stopping  points  for  which  I^(k,s)  =  L2(k,s). 
If  the  largest  (smallest)  lower  (upper)  stopping  point  is  called  a),  (resp.  b^), 
then 


t  =  min{k>l:  S.< a-  or  S  >  bjJ 

k  x 

and  it  is  optimal  to  decide  that  p  <  pg  if  and  only  if  S^-  < 


The  several  tables  which  accompany  this  letter  describe  the  optimal 
sampling  plan  in  a  special  case  in  which  m  =  12,  hp  is  a  beta  density  with 
parameters  a  =  6  and  b  =  2,  w  =  3/4,  pg  =  3/4,  Cp  =  60,  and  C2  =  180.  Here  the 
ratio  C2/Cp  =  3  equates  the  seriousness  of  deciding  that  p  <  pg  when  p  >  p 
with  that  of  deciding  that  p  >  pg  when  pg  -  p  =  1/3;  and  the  magnitudes  of  Cp 
and  C2  were  chosen  to  make  it  optimal  to  take  up  to  about  12  observations. 

I  believe  that  this  is  consistent  with  the  power  and  sample  size  requirements 
discussed  earlier.  In  a  certain  sense,  these  values  of  Cp  and  C2  are 

implicit  in  those  requirenents. 


Table  1  lists  the  boundaries  and  b^  of  the  optimal  test.  These 
boundaries  are  remarkably  insensitive  to  a+b.  I  got  nearly  the  same  values 
when  a  =  9  and  b  =  3.  Table  2  lists  an  ad  hoc  modification  of  the  optimal 
boundaries  which  takes  account  of  th^  economies  of  testing  items  in  groups  of 
three.  Table  3  gives  the  posterior  probability  that  p  >  pg  for  each  possible 
outcome,  using  the  adhoc  boundaries.  It  clearly  echibits  the  following 
qualitative  feature  of  the  test:  if  the  results  of  the  first  six  tests  this 
year  are  consistent  with  last  year's  results,  then  further  testing  is  rot 
optimal.  Table  4  gives  the  frequentist  properties  of  the  adhoc  test,  the  power 
function  and  expected  sample  size  as  a  function  of  p.  Observe  that  the  maximum 
expected  sample  size  is  substantially  snaller  than  that  of  the  adhoc  test;  and 
recall  the  crucial  role  of  the  rraximum  in  determining  the  number  of  items  which 
must  be  purchased  for  testing. 


TABLE  1:  AN  OPTIMAL  BOUNDARY 


Parameters : 

m=k,  a=l 

,  b=2,  w=3/4, 

p-3/4.  Cl =60 ,  C2=180 

k 

ak 

bk 

1 

__ 

l 

2 

0 

2 

3 

0 

3 

4 

1 

4 

5 

2 

4 

6 

2 

5 

7 

3 

6 

8 

4 

6 

9 

5 

7 

10 

5 

7 

11 

6 

8 

12 

7 

8 

TABLE  #2 

:  A  MODIFIED 

BOUNDARY 

k 

ak 

bk 

1 

__ 

2 

— 

_ 

3 

0 

3 

4 

0 

4 

5 

0 

5 

6 

1 

5 

7 

2 

6 

8 

3 

6 

9 

4 

7 

10 

5 

7 

11 

6 

8 

12 

7 

8 

TABLE  #3: 

POSSIBLE 

OUTCOMES  WITH 

MODIFIED  BOUNDARY 

k 

Sk 

P(p^  po) 

3 

0 

.0251 

6 

1 

.0084 

6 

2 

.0507 

7 

3 

.0813 

8 

4 

.1211 

9 

5 

.1634 

11 

6 

.1185 

12 

7 

.1546 

12 

8 

.3111 

10 

7 

.4543 

8 

6 

.5183 

6 

5 

.6517 

3 

3 

.7450 

TABLE  #4:  FREQUENTIST  PROPERTIES 


£ 

BETA 

MEAN 

VAR 

.05 

.9999 

3.4575 

1.281 

.  1 

.999 

3.8288 

2.4702 

.15 

.9983 

4.4161 

3.5485 

.2 

.9903 

4.8134 

4.5345 

.  25 

.9788 

5.4154 

5.43 

.3 

.9582 

5.8102 

6.2305 

.35 

.9244 

6.3797 

6.8348 

.40 

.  8728 

6.7887 

7.559 

45 


BUOO 


7.1384 


8.1442 


Comments  by  DW : 

With  this  note  Woodroofe  completes  the  transition  from  Wald's 
classic  treatment  to  a  Bayesian  approach.  The  use  of  a  prior 
probability  which  is  a  mix  of  two  hypotheses  is  in  part  an  attempt  to 
address  the  criticism  that  priors  can  become  too  sharply  peaked, 
neglecting  the  potential  staleness  of  old  data.  One  might  still  ask 
whether  there  should  be  an  upper  limit  to  the  value  of  k  used  in  the 
prior. 

The  loss  functions  included  in  this  section  are  representative, 
rather  than  my  recommendation.  The  variable  called  po  in  the 
functions  LI  and  L2  could  have  different  values  in  the  two  cases. 


Chapter  V 

Other  Stopping  Criteria 


A  possible  argument  for  small  test  sizes  may  arise  after  all 
missiles  have  been  bought:  any  test  reduces  the  potential  tactical 
inventory.  The  decision  criterion  is  unfortunately  not  unique.  This 
chapter  discusses  a  few  examples. 

Section  1.  Utility  as  a  Criterion 

Let  <£  (p  i  be  the  posterior  probability  distribution 

of  p,  given  s  "equivalent"  successes  and  f  "equivalent"  failures  on 
which  to  base  a  prediction.  Let  U  (N,p)  be  the  "utility"  of  an 
inventory  of  N  missiles  of  reliability  p.  The  estimate  of  the 
utility  of  the  inventory  is  then 


Now  perform  a  test:  N  goes  to  N— 1 j  with  probability  p, 
s  goes  to  s+1 ;  and  with  probability  l-p(  f  goes  to  f+1. 

After  the  test  the  utility  is 

U (m-  \)~  \  0 ■+  • 

The  criterion  is:  Is  U(N-1)>U(N)? 


Examples  of  utility  functions  are: 


Np  (expected  targets  killed); 
-Np(l-p)  (uncertainty  is  reduced); 


N-T/P  (excess  inventory,  where  T  is  size  of 
_  target  list); 

T(_l-  (expected  damage) ; 

T[l- 

(b*largest  integer  in 
fractional  part;  this  reduces  to  Np  for  small  N, 
damage  for  large  N). 


critical 


N/T ;  a*=N/T-b  is 
goes  to  expected 


the 


Clearly  there  is  a  similarity  between  this  method  and  that  in 
Secion  4  of  the  previous  chapter. 

Section  2 .  Information  as  a  Criterion 

Another  criterion  would  be  the  information  the  decision  maker 
gains  from  the  test  about  the  posterior  distribution  of  p.  This 
would  be  applicable  when  no  single  utility  function  can  be  agreed  on. 
An  example  is  the  Kullback-Leibler  information  measure  on  two 
probability  density  functions 


FI  and  F2  (Reference  18): 

r(p,  S^af- 

It  can  be  applied  to  the  current  problem  by  defining  FI  and  F2 
respectively  as  the  posterior  and  prior  density  functions  for  p. 

Shannon’s  information  measure  S(F1,F2)  is  the  expectation  value  of 
I(F1,F2)  over  the  observed  values  of  success  and  failures. 

To  illustrate,  we  may  identify  F2  with  expression  1.6  from 
Chapter  I: 

and  FI  with  expression  1.8: 

f,  (jf>  vs,+Sv"  ('- 

so  that  log  F1/F2  is  — j 

r  c^)F(?^or  cv  y 

-  (2.  +  -fv  ^6- 


where  C  is  the  logarithm  of  the  gamma-function  combination  in  curly 
braces,  all  independent  of  p.  Noting  that 

Af 

h/  \  — 

and  letting  T  W—  C(i)  &  1=  the  logarithmic  derivative  of  the 
gamma  function,  the  expression  for  I(F1,F2)  reduces  to 


i 


r  (f,  c  - 


Then 


Consider  now  the  case  where  s2=n2=l  (a  single  successful  trial). 


Isr 


In  the  alternative  case  wher  S2«=0,n2*l  (a  single  unsuccessful  trial) 


Ip  fat)] 


and  the  Shannon  information  is 


s,  is+£,rf 


As  this  never  goes  to  zero  (for  finite  nlj,  the  cost  of  this 
information  must  be  balanced  against  the  use  made  of  it. 

1  have  not  yet  found  a  way  to  apply  this  criterion  to  the 
Pershing  testing  problem. 


Chapter  VI 
Conclusion 


I  return  now  to  the  tasking  from  the  Under  Secretary  of  the 
Army,  as  given  in  the  opening  of  this  memorandum.  The  mathematical 
methods  of  sequential  analysis  proposed  here  for  estimating 
reliability  changes  possess  a  rigor  not  found  in  the  Army's  current 
method,  and  make  clear  the  risks  in  following  their  prescription. 

They  provide  a  basis  for  reducing  the  size  of  an  annual  test  and  so 
reducing  too  the  cost  of  a  testing  program.  Indeed,  they  even 
challenge  the  need  for  an  annual  report,  and  suggest  that  the 
interval  between  reports  can  be  enlarged  (e.g.,  to  two  years)  with  no 
increase  in  risk  to  management.  They  do  not,  however,  encompass  a 
variety  of  other  issues  which  are  fundamentally  operational  in 
nature:  firings  to  support  training,  alternate  uses  of  inventory, 

system  life.  These  must  be  the  subject  of  further  investigation. 

Readers  of  this  report  may  be  disappointed  that  such  very 
different  approaches  to  the  stopping  problem  have  been  presented  in 
the  foregoing  chapters.  I  observe  that  such  a  seemingly  simple 
problem  has  apparently  not  been  hitherto  subject  to  the  scrutiny  it 
deserves,  and  that  it  is  comforting  that  two  separate  investigations 
have  reached  similar  conclusions. 

I  see  ultimately  more  promise  in  the  methods  proposed  in  Chapter 
IV,  but  would  recommend  that  those  of  Chapters  III  and  IV  be  applied  t 
to  Pershing  using  the  best  available  data  so  that  a  refined  test 
program  can  be  determined.  In  Chapter  III  is  proposed  the 
application,  as  yet  unexplored,  of  Kalman  filtering  techniques  to 
this  problem.  This  research  merits  monitoring,  if  not  support. 
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Appendix  D 
HP-41  Programs 


The  HP-41  handheld  calculator  is  slow  but  remarkably  powerful. 
For  example,  a  program  listing  for  the  standard  Fast  Fourier 
Transform  (FFT)  algorithm  is  no  lengthier  than  that  for  a  FORTRAN 
version  and  because  of  some  quirks  of  the  HP-41,  the  program  is  in 
some  ways  more  efficient.  With  a  56-bit  word,  numerical  accuracy  is 
higher  than  in  most  personal  computers,  and  so  round-off  problems  are 
slower  to  arise. 

Reported  in  this  appendix  are  a  set  of  programs  written  for  this 
study.  Their  original  purposes  were  to  give  or  to  verify  solutions, 
but  they  have  two  additional  values  justifying  their  inclusion  here: 
they  demonstrate  that  the  mathematics  called  upon  is  not  intractible 
and  can  be  packaged  small,  and  they  may  be  useful  as  is  to  others 
working  the  same  or  related  problems. 

The  first  group  provde  solutions  to  Equations  1.9  and  1.11  and 
thus  can  be  considered  a  proper  means  of  getting  the  answers  wrongly 
sought  via  Fishei's  Exact  Test.  The  versions  given  are  lengthy  but 
are  relatively  robust  to  the  accumulation  of  round-off  errors. 
Included  is  the  program  PII,  written  to  be  a  model  for  and  to  verify 
calculations  of  Singpurwalla  and  Launer. 

The  second  group  provide  handy  means  of  exploring  Woodroofe's 
treatment  of  sequential  analysis.  ET  provide  solutions  to  Equations 
1  and  2  of  Chapter  III,  Sec  1.  BND  provides  Wald's  and  Woodroofe's 
boundaries  of  the  region  of  test  continuation;  and  MW  permits 
computation  of  a  number  of  properties  of  a  test  plan  defined  by  BND. 
LOP  computes  boundaries  using  the  Bayesian  method  of  Chapter  III, 

Sec .  4 . 

Not  included  is  a  package  of  routines  which  manipulate  truncated 
Taylor  series  and  was  used  to  compute  the  expansion  of  D(n,m)  given 
in  Eq  4.4.  This  is  available  from  the  author. 

The  memory  requirements  of  an  HP-41CV  or  CX  are  needed,  and  if 
it  is  not  the  CX  version,  then  an  Extended  Functions  module  (XF)  with 
its  Expanded  Memory.  The  occasional  use  of  Synthetic  Programming  can 
be  circumscribed,  or  if  the  programs  are  identical  to  those  listed 
here,  they  should  run  on  any  version  of  the  HP-41  with  adequate 
memory  and  the  XF  module. 


JCS+  Implements  Eq.1.9  and  DA+  Eq.1.11. 


They  call  for  inputs  and  report  the  value  of  the  integral  as 
"CL*="  for  Confidence  Level.  The  plus  sign  means  there  are  no 
subtractions  in  the  algorithm,  hence  less  round-off  error. 


PII  Implements  Eqs.4-6  of  Section  III. 3. 

Entering  at  LBL  A  leads  to  an  evaluation  of  ^  and  at  LBLT&  to 
evaluation  of  (3  .  Lines  51-62  clear  a  block  of  registers,  using 

program  BC  in  a  module  called  PPC  ROM.  This  can  be  replaced  by 
ordinary  coding.  If  Flag  02  is  set,  then  the  summation  sign  in  Eq.4 
or  5  is  ignored;  only  a  single  term  is  considered.  Subroutines  1,  2, 
and  13  are  the  core  of  algorithm. 


ET 


Solves  Eqs.  1  and  2  of  Section  IV. 1. 
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Calls  for  N,  c,  and  p  (unadjusted  values  will  be  used  as  is). 

Memory  utilization  keyed  to  that  in  MW:  N,  c,  and  p  in  same 
registers . 


MW 


Requires  two  files  in  Extended  Memory  named  Am  and  Bm  where  m  is 
a  number  provided  in  response  to  query  "FILE#?"  or  is  already  stored 
in  register  19.  (Routine  BND  may  have  been  used  to  create  these 
files .  ) 

Start  program  at  line  1  or  at  LBL  E;  line  one  to  provide/revise 
the  value  of  N,  the  maximum  number  of  tests.  At  E,  provide  "p"  and 
"FILE#."  If  RAD-DEG  selection  set  to  RAD,  program  computes  and 
reports  G(k)  as  required  by  Section  IV. 3;  if  set  to  DEG,  this  is 
ignored . 

Program  reports  (p),  E(t),  and  a  (p)  (which  in  effect 
interchanges  meaning  of  "reliable"  and  "unr eliable*)  .  Sect  IV. 1. 


.  LBL  B  produces  output  stating  "bi/i  =  cumulative  probability  of 
sufficient  failures  to  halt."  Accumulates  probability  of  exit 
passing  clockwise  around  boundary.  If  there  are  several  points  on 
boundary  at  N=N  max,  then  these  are  labeled  F.  Then  program 
continues  along  "a"  boundary. 

LBL  C  does  the  same  as  LBL  B  but  counterclockwise. 

LOP 

To  meet  the  goals  of  Section  IV. 4.  Computes  the  boundary 
conditions  for  continued  testing,  based  on  the  loss  functions  LI  and 
L2  (which  can  have  associated  with  them  different  criteria  PI  and  P2, 
as  well  as  cost  factors  Cl  and  C2). 

Program  invites  all  necessary  input  insertion/revision/ 
verification,  and  then  constructs  a  diagram  cf  the  operating  space. 

To  conserve  space  this  pattern  is  stored  as  packed  binary  data  (a  la 
flags).  LBL  J  provides  a  visualization  of  this  pattern,  for  display 
or  printing  (see  figures  below).  This  algorithm  has  also  been  run  on 
a  Commodore  for  verification. 

Routines  6  and  7  support  generation  of  loss  functions  L t  and  Lj£ 
If  others  are  chosen,  these  must  be  rewritten  along  with  some  of 
Routine  2  (lines  57-100). 

BND 

Develops  the  boundaries  to  be  used  in  MV,  by  Wald's  and 
Woodroofe's  methods.  Input  called  for:  P0,  PI,  a,  and  b  (later,  m). 

0<P0<P1<  1.  Level  of  test  «=  a.  Probability  of  Type  II  error  =  b 

(P^2-  PI).  Ho:  p^-po.  (Section  IV. 2).  M  is  number  of  tests. 

Lines  1-85:  Wald's  methods,  an*  and  bn1  reported  out. 

86-156:  Woodroofe's  modification. 

157-END:  Subroutine  E.  Calls  for  a  file  number  k;  then  stores 

Woodroofe's  boundary  numbers  a  n  and  bp,  in  files 
AK  and  BK.  If  Flag  25  is  clear  to  start,  program 
halts  if  attempt  is  made  to  overwrite  existing  file. 
Set  the  Flag  to  permit  overwriting. 
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APPENDIX  A 


An  Algorithm,  A  Computer  Code,  and  A  User’s  Guide,  for 
a  Bayesian  Binomial  Hypothesis  Testing  Procedure 


A . 1 .  INTRODUCTION 

In  the  Bayesian  binomial  hypothesis  testing  procedure,  we  need 
to  find  the  pair  (n^,x*)  such  that  [see  Equations  (4)  and  (5)]: 


1  x* 


/  1 

jn  J-0 


n 

r 

u  J 


n  -j 

Pt  (1  “  Pt^  C  8(Pt)dpt  <  a 


and 


1  x*  ( 

/t  n  .  n  -j 

l  1  (Pt  ~  A)J  (1  -  p  +  A)  g(Pt)dpt  >  1  -  B  , 

j=0  U  J 


where 


,  v  r (y+6)  y-1  N  6-1 

8(pt}  r(y)r(6)  pt  (1  "  pt) 


The  above  inequalities  can  be  rewritten  as: 
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(8A) 


(9A) 


>  1  -  B  , 
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19 


where 

1 

B(A,1;  r,s)  =  J  1  (1  -  pt)S  1  dpt  . 

A 

A  computer  code  designed  to  obtain  the  smallest  values  of  nt  , 
x*  subject  to  the  two  inequalities  (8A)  and  (9A)  ,  based  on  an  enumera¬ 
tion  procedure  discussed  next,  is  obtained. 

A. 2  DESCRIPTION  OF  THE  ENUMERATION  PROCEDURE 

The  enumeration  procedure  exploits  the  fact  that  both 
and  g2(xt>nt>  are  increasing  functions  of  xfc  if  n^  is  fixed.  The 
procedure  starts  with  some  initial  value  of  nt  ,  say  n^  ,  and  finds 
the  largest  xt  such  that  g^Cx^.n^)  <  a  .  Once  such  an  x^_  ,  say  x^  , 
is  found,  it  is  guaranteed  that  the  first  inequality  will  be  satisfied 

for  values  of  xt  smaller  than  x^  .  The  procedure  then  tries  to 

find  an  x^  smaller  than  x^  such  that  g2(xt>nt)  ^  1  -  B  •  If  such 

an  xt  does  not  exist,  the  value  of  nt  is  increased  by  one  and  the 

procedure  starts  all  over  again.  As  n^  increases,  the  procedure  finds 
the  smallest  values  of  nt  and  xt  satisfying  both  inequalities.  The 
flow  chart  for  this  enumeration  procedure  is  presented  in  Figure  A.l. 


A. 3  THE  COMPUTER  CODE 

The  program  requires  certain  JCL  cards  and  a  user  input  of  some 
parameters . 
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Figure  A.l.  Flow  chart  for 
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A. 3.1  Input  Specifications 

The  cards  should  be  arranged  in  accordance  with  Figure  A. 2;  each 
card  will  be  explained  individually. 

Job  Card  and  JCL  Cards:  The  standard  job  card  is  used  and  so  are 
the  following  JCL  cards: 

//0EXEC0FORG2 
//FORT . SYSIN0DD 
//GO.SYSLIB0DD 

//^0(iDDt5|6^DSN=GWU.IMSL.V9.DLOAD,DISP=SHR 

//GO.SYSINW«M5DD0M»&* 

where  the  character  "J6"  indicates  a  blank  space.  The  first  two  JCL 
cards  immediately  follow  the  job  card.  The  remaining  JCL  cards  are 
placed  after  the  program  and  just  before  the  input  information  card. 

The  fourth  JCL  card  is  needed  to  use  the  IMSL  subroutines  on  an  IBM 
machine . 

Input  Information  Card — DEL,  SGM,  SDEL,  ALF,  BETA,  NT:  This  card 
contains  sorted  input  information,  DEL,  SGM,  and  SDEL,  which  are  the 
parameters  A  ,  y  ,  and  6  in  Equations  (8A)  and  (9A) ;  ALF  and  BETA  are 
the  right-hand  side  parameters  a  and  8  in  these  inequalities.  These 
parameters  are  specified  in  format  F10.5.  The  input  NT  is  the  initial 
value  of  nt  selected,  and  is  in  14  format.  Usually,  this  value  is  one. 
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Figure  A. 2.  Card  deck  structure. 


A. 3. 2  Interpretation  of  Output 

The  program  uses  an  iterative  scheme  and  evaluates  ®x^Xt,nt^ 
and  g9(x  ,n  )  for  different  values  of  x  adn  n  .  On  the  output, 

/  t  t  U  U 

the  values  of  g^Cx^.n^  and  g2(xt>nt)  are  printed  as 

FIRST  CONST  = 

SECOND  CONST  - 


for  different  values  of  x  and  nc  . 


The  solution  of  the  problem,  that  is,  the  smallest  values  of  x 
and  n  satisfying  the  inequalities  (8A)  and  (9A) ,  are  printed  in  the 
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last  line  of  the  output  as 

X  =  N  = 

Sample  output  is  presented  in  Table  A.l. 

The  smallest  values  of  x£  and  n£  satisfying  the  inequalities 
(8A)  and  (9A)  are  X  =  10  and  N  =  15  .  In  this  example,  the  values 
of  the  parameters  are  A  =  0.25  ,  y  =  106  ,  6  =  19  ,  a  =  0 . 10  ,  and 
B  =  0.25  .  The  initial  value  of  n  is  one. 

The  listing  of  the  program  is  given  in  Appendix  B. 
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TABLF  A.l 
Sample  Output 


FIRST 

COM 3T= 

0.000  00 

FIRS? 

CON  ST  = 

o. oooo : 

FIRST 

C  0  i  S  v  = 

0.00000 

FIRST 

CONST= 

0.0  00  -2 

FIRST 

CO  N  3T  = 

0.00010 

FIRST 

CON  3 T  = 

0.00133 

FIRST 

CO  N  ST  = 

0. 0073* 

FIRST 

CO  N  ST  = 

0.  0  3 1  ii  0 

FIRST 

CON  3T  = 

0  .  1  0  S  2  0 

CCCONO  C 0 V 3 T  = 

0. 55275 

FIRST 

CO  iST  = 

0. 0  00  OS 

FIRST 

C0N2I= 

0.00000 

FIRST 

CON  ST  = 

i .  o  oo :  j 

FT  PS7 

CONS?= 

o.ooou 

firs: 

C  )  N  3  ?  = 

0.00005 

RTFS? 

co  ;s?= 

U  0  003  1 

F  I  F  5  T 

coTsr= 

0. 00245 

FIRST 

CO  iST  = 

0.01  IN? 

FIRST 

C  0  N  T"  = 

•;  .  0  4  3  7  0 

FIRST 

const* 

0.1 32: 0 

~:ccs: 

>  - ) ;  s  t  = 

0  .  5  5  *•  3  ~ 

RT  :?ST 

CO  N  ST - 

0.000.. 

i ■  T  ?cy 

CON  rt  = 

0.0  0  0 

F  I  HCT 

00131= 

o .  o  o  o;  o 

FIRST 

CO  N  ST  = 

o. ooo: ) 

FIRST 

CO**  3T  = 

0.000*2 

FIRST 

CON  31  = 

0.00012 

FIRST 

C  )V.ST  = 

0 . 0  0  0  r  p 

FT  RST 

CON  Zm  = 

J  •  0  u  4  1  '0 

FIRST 

C  J  *1  3  T  = 

0  .  N  1  7  1  7 

FIRST 

CONST = 

0. T  3357 

FIRST 

C  0  V  S  T  = 

0.162* 5 

'ICON D 

C  .0  N  3 '”  = 

0.71433 

"  r  psp 

CUN’  3T  = 

0.  .000  0 

FIRST 

CON  ST  = 

0. 000.00 

firs: 

CON  ST  = 

0.00000 

FIRST 

:jnst  = 

0.0000 .0 

FIRST 

CON  ?T- 

0 . 00002 

FIRST 

con  sr  = 

3.30004 

FIRST 

CON  ST  = 

^  -1  -0  c 

•J  •  -  -  U  t  7 

FIRST 

CON  ST  = 

o .  o  o  i  n  i 

Fipsr 

CO  i  3”  = 

0  .  0  0  5  'i  3 

FIRS7' 

v,  0  N  .  T  = 

’•02422 

FIRST 

■CON  3*  = 

3  .  p  7  3  7  “■ 

FUST 

0  0  N  3  T  = 

0.10037 

FT  CO'  D 

cons  : - 

.  .  7«  ?f.  u 

if co ;  o 

CONST- 

7 . 30272 

'/ 

=  10.  uo 

■>  o  v  - 

>  /  <  - 

xr  = 

C.  2 

s  r = 

12 

xr= 

1.0 

N  T  = 

12 

XT  = 

2.0 

n  r  = 

12 

xr= 

3.0 

NT  = 

12 

xr= 

14. C 

NT  = 

12 

xr  = 

5.  0 

n  r  = 

12 

xr= 

?  .0 

vj  y  = 

12 

xr  = 

7.0 

i:t  = 

12 

xr  = 

3.0 

:nt  = 

1  2 

y  ?= 

=  7.0. 

NT= 

12 

xr= 

Cl 

-w  « 

^  T  — 

1  3 

:<?  = 

uo 

>.  r  = 

1  3 

y?= 

NT  = 

13 

y  r  = 

3.  C 

N  t  = 

1  3 

vr  = 

14 . 0 

N  T  = 

1  3 

yr  = 

3.2 

•*  T  = 

1  3 

>:  r  = 

*.C 

:it  = 

13 

>:r  = 

7.0 

N  V  = 

1  3 

•  '  T‘ 

-.C 

NT  = 

13 

FT  = 

0.0 

y  t= 

1  3 

’  T  = 

e. : 

u  r  = 

1  3 

/  ”  - 

'  r 

•  4. 

n  r  = 

1  4 

■  r  = 

1.0 

N‘T  = 

1  4 

xr  = 

2.0 

•f  r»  _ 
t  .  — 

1  4 

X  T  = 

.3.  0 

ST  = 

14 

Y  T  _ 

14.0 

«?  = 

14 

xr  = 

5.0 

n  r  = 

1  4 

V  **  — 

5.2 

n  r= 

1  4 

XT  = 

7.0 

V  T  = 

1  4 

x  r  = 

-..o 

ir  = 

1  4 

xr  = 

0.7 

NT  = 

14 

vv- 

10.  0 

»T  = 

14 

<T  = 

3.2 

n  r  = 

1  4 

yr= 

0.0 

NT  = 

15 

XT  = 

1.0 

nt  = 

15 

X  T  = 

2 .  C 

vr  = 

1  5 

X  T  = 

3.  0 

N  T  = 

15 

y  r  = 

4.2 

S'  I  = 

15 

yp  = 

5.2 

n  r  = 

1  5 

>  T  = 

f  .2. 

V  r- 

15 

\r  = 

7.  r 

«  r  = 

1  5 

y  r  - 

C.  2 

n  r  = 

1  5 

aT  = 

0.  0 

N  T  = 

1  5 

\r  = 

10.0 

r= 

15 

y  r  - 

11.0 

s  r  = 

1  3 

■  n  _ , 

17.2 

N  T- 

15 

’  T= 

9.0 

VT  = 

15 

15 


APPENDIX  B 


A  Listing  of  the  Program  for  a  Bayesian 
Binomial  Hypothesis  Testing  Procedure 
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LIM-1,5 

,1-TEST 

//  EXEC  F0RX2 
//FORT. SYS  IN  DD  * 

IMPLICIT  REAL*8  (A-H.O-Z) 

INTEGER  IER 

READ  (5,10)  DEL.SGM.SDEL, ALF .BETA.NT 

10  FORMAT (5F 10.5, I  A) 

BET-1 .O-BETA 
Xl-DEL 

X2-1  .0 

C  WE  START  THE  ALGORITHM  BY  INITIATING  XT  AS  ZERO 
W1-SGM 
W2-SDEL 
A1-W1 
B1-W2 

CALL  FACT]  (A1.B1.S0N) 

W-SON 

11  XT-0.0 
WNT-NT 
WA-WNT+SDEL 
TA1-SGM 
TB1-WA 

CALL  FACT2  (TA1 ,TB1 ,TERS) 

PAR-TERS 
CO 1-W*PAR 

C  THIS  IS  THE  VALUE  WHEN  XT  IS  ZERO 

C  NOW  WE  COMPUTE  THE  VALUE  G1  WHEN  XT  IS  OTHER  THAN  ZERO. 

301  IXT-XT 
T0T-C01 

IF  (XT.EQ.O.O)  GO  TO  1001 
DO  1000  1=1, IXT 
R  I  - 1 

P1-W1+RI 

P2-WA-RI 

TA1-P1 

TB1-P2 

CALL  FACT2  (TA1 ,TB1 ,TERS) 

P3-WNT+1 .0 
PA-P3-RI 
P5-RI+1 .0 

Z-  (DGAMMA  (P3) )  /  ( (DGAMMA  (PA) )  *  (OGAMMA  (P5) ) ) 

P-TERS 

TOT-TOT+  (P*Z*W) 

1000  CONTINUE 

1001  G1-T0T 

WRITE  (6,60)  Gl.XT.NT 

60  FORMAT  (5X, 'FIRST  CONST- ' , F 10 . 5, 5X . ' XT- • , F5 . 1 , 5X, ' NT- ',1  A) 
C  SO  WE  COMPUTEO  THE  VALUE  OF  FIRST  CONSTRAINT 
IF  (Gl.GT.ALF)  GO  TO  333 
IF (XT.EQ.NT)  GO  TO  380 
XT-XT+1 .0 
GO  TO  301 
333  XT-XT-1.0 
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IF (XT. LT. 0.0)  GO  TO  999 
C  OTHERWISE  WE  GO  AND  CALCULATE  G2 
380  WW*W* (DEL**WNT) 

C  NOW  COMPUTE  THE  VALUE  WHEN  XT  IS  ZERO, THAT  IS  J  IS  ZERO. 

C  WHEN  J  IS  ZERO  L  IS  ZERO 

C  WHEN  J  IS'  ZERO.M  GOES  FROM  ZERO  TO  NT  AND  L  IS  ALWAYS  ZERO  IN  THIS  CASE 
C  FIRST  CONSIDER  THE  CASE  WHERE  WHEN  M  IS  ZERO 
A*W1 
B-W2 
TA1-W1 
TB1-W2 

CALL  FACT2  (TA1 ,TB1 ,TERS) 

CALL  MDBETA  (XI .A.B.PI , I ER) 

CALL  MDBETA  (X2 , A, B , P2 , I ER) 

Y*TERS 

VALO*  (P2-P1) *Y 
SUM-VALO 

C  NOW  CONSIDER  THE  CASES  WHERE  M  IS  ONE  TO  NT. 

DO  1500  M= 1 , NT 

A-W1 

BM*M 

BM1-WNT+1 .0 
BM2-WNT-BM+1 .0 
BM3=BM+1 .0 

BMCOM-DGAMMA  (BM1) / ( (DGAMMA (BM2) )  * (DGAMMA (BM3) ) ) 

BFAC*  (DEL**  (-BM) ) *BMCOM 

B-W2+BM 

TA1-W1 

TB1-B 

CALL  FACT2  (TA1 ,TB1 ,TERS) 

CALL  MDBETA  (XI , A.B.PI , I ER) 

CALL  MDBETA  (X2 , A, B , P2 . I ER) 

Y*TERS 

VAL*=  (P2-P1)  *Y*BFAC 
SUM=SUM+VAL 
1500  CONTINUE 
JXT-XT 
RJSUM-SUM 

C  IF  XT  IS  ZERO  WE  HAVE  ONLY  THE  ABOVE  TERM 
IF  (XT.EQ.O.O)  GO  TO  2001 
DO  2000  J-l.JXT 
C  THIS  IS  THE  MOST  OUTER  SUM 
RJ*J 

RJ1-WNT+1 .0 
RJ2-WNT-RJ+1 .0 
RJ3-RJ-M  .0 

COMB  J-  (OGAMMA  (RJ 1) )  /  ( (DGAMMA  (RJ2) )  *  (DGAMMA  (R  J  3) ) ) 

C  NOW  L  IS  FROM  ZERO  TO  J. AGAIN  CONSIDER  THE  CASE  WHERE  L  IS  ZERO 
LP«(-1)**J 
PL-LP 

C  NOTE  WHEN  L  IS  ZERO  M  GOES  FROM  ZERO  TO  NT-J 
LJL-NT-J 

IF  vLJL.EQ.O)  GO  TO  2101 
00  2100  M-l.LJL 
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RRM-M 

RRM1-WNT-RJ+1 .0 
RRM2-WNT-RJ-RRM+1 .0 
RRM3-RRM+1 .0 

RC0M-  (DGAMMA (RRM1) ) / ( (DGAMMA (RRM2) )  * (DGAMMA (RRM3) ) ) 

FFAC-  (DEL**  (-RRM) )  *RCOM 

A-SGM 

B-RRM+SDEL 

TA1-A 

TB1-B 

CALL  FACT2  (TA 1 ,TB 1 ,TERS) 

CALL  MDBETA  (X 1 , A, B , P 1 , I ER) 

CALL  MDBETA  (X2, A.B.P2, IER) 

Y-TERS 

VALM-  (P2-P 1 ) *F  F  AC*Y 
VALO-VALO+VALM 

2100  CONTINUE 

2101  RLSUM-VALO*PL 

C  THIS  IS  THE  VALUE  WHEN  L  IS  ZERO 

C  NOW  WE  WANT  TO  CONSIDER  L  FROM  1  TO  J.THIS  IS  THE  SECOND  SUM 
DO  2500  L-l.J 
RL-L 

RLl-RJ-RL+1  .0 
RL2-RL+1 .0 

COMBL-  (DGAMMA  (R J 3) )  /  ( (DGAMMA  (RL  1) )  *  (DGAMMA  (RL2) ) ) 

LPL«(-1)**(J-L) 

FLP-LPL 

POWER-DEL**  (-RL) 

FACL-FLP*COMBL*POWER 

C  NOW  SHOULD  CONSIDER  M  LOOP  AGAIN. NOW  M  IS  FROM  ZERO  TO  NT-J  FOR  GIVEN  L 
C  START  WITH  MIS  ZERO 
A-RL+SGM 
B-SDEL 

CALL  MDBETA  (X 1  .  A .  B  .  P 1  ,  I  ER) 

CALL  MDBETA  (X2.A.B.P2, IER) 

TAI-A 

TB1-B 

CALL  FACT2  (TA 1 ,TB 1 ,TERS) 

Y-TERS 

VAL-  (P2-P1) *Y 

RMSUM-VAL 

LL-NT-J 

IF  (LL.EQ.O'  GO  TO  3001 

DO  3000  M-l.LL 

RM-M 

RM1-WNT-RJ+1.0 
RM2-WNT-RJ-RM+1 .0 
RM3-RM+1.0 

C0M8M- (DGAMMA (RM1) )  / ( (DGAMMA (RM2) ) *  (DGAMMA  (RM3) ) ) 

FACM-  (DEL**  (-RM) )  *  (COMBM) 

A-RL+SGM 

B-RM+SDEL 

CALL  MDBETA  (XI ,A,B,P1 , IER) 

CALL  MDBETA  (X2, A, B,P2, IER) 
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TA1-A 

TB1-B 

CALL  FACT2 (TA1 ,TB1 ,TERS) 

Y-TERS 

VAL- (P2-P1) *FACM*Y 

rmsum-rmsum+val 

3000  CONT I NUE 

3001  RRSUM-RMSUM 

C  THE  MOST  INNER  LOOP  IS  FINISHED. 

RLSUM* (FACL*RRSUM) +RLSUM 
C  THIS  IS  THE  SUM  FOR  L  LOOP 
2500  CONTINUE 
C  L  LOOP  IS  FINISHED 

C  NOW  FINISH  J  LOOP. THE  MOST  OUTER  LOOP. 

RJSUM* (COMBJ*RLSUM) +RJSUM 

2000  CONTINUE 

C  SO  WE  EVALUATED  G2. 

2001  G2«RJSUM*WW 

WRITE  (6,61)  G2.XT.NT 

61  FORMAT (5X, 'SECOND  CONST* ' . F 10 . 5 . 5X , ' XT- ' , F5. 1 , 5X . ' NT- 1 , I U) 

IF  (G2.LT.BET)  GO  TO  999 
777  IF  (XT.LT.l .0)  GO  TO  888 
XT-XT-1.0 
C  CHECK  G2  AGAIN. 

WW-W*  (DEL**WNT) 

C  NOW  COMPUTE  THE  VALUE  WHEN  XT  IS  ZERO, THAT  IS  J  IS  ZERO. 

C  WHEN  J  IS  ZERO  L  IS  ZERO. 

C  WHEN  J  IS  ZERO , M  GOES  FROM  ZERO  TO  NT  AND  L  IS  ALWAYS  ZERO  IN  THIS  CASE 
C  FIRST  CONSIDER  THE  CASE  WHERE  WHEN  M  IS  ZERO 
A-Wl 
B-W2 

CALL  MDBETA (X 1  .  A ,  B ,  P 1 ,  I  ER) 

CALL  MDBETA (X2 , A, B.P2, IER) 

TA1-A 

TB1-B 

CALL  FACT2  (TA 1 ,TB  1  ,TER5) 

Y-TERS 

VALO-  (P2-P1) *Y 
SUM-VALO 

C  NOW  CONSIDER  THE  CASES  WHERE  M  IS  ONE  TO  NT. 

DO  1501  M-l.NT 

A-Wl 

BM-M 

BM1-WNT+1 .0 
BM2-WNT-BM+1 .0 
BM3-BM+1.0 

BMCOM-DGAMMA  (BM1) /  ( (DGAMMA (BM2) ) * (DGAMMA (BM3) ) ) 

BFAC- (DEL** (-BM) ) *BMCOM 

B-W2+BM 

TA1-A 

TB1-B 

CALL  FACT2  (TA 1 ,TB 1 ,TERS) 

CALL  MDBETA (XI ,A,B,P1 , IER) 

CALL  MDBETA (X2 , A , B, P2 , I ER) 
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Y-TERS 

VAL-  (P2-P 1 ) *Y*BF  AC 
SUM-SUM+VAL 
1501  CONTINUE 
JXT-XT 
RJSUM-SUM 

C  IF  XT  IS  ZERO  WE  HAVE  ONLY  THE  ABOVE  TERM 
IF  (XT.EQ.O.O)  GO  TO  2011 
DO  5000  J-l.JXT 
C  THIS  IS  THE  MOST  OUTER  SUM 
RJ-J 

RJ1-WNT+1 .0 
RJ2-WNT-RJ+1 .0 
RJ3-RJ+1 .0 

COMBJ-  (DGAMMA  (RJ 1 ) )  /  ( (DGAMMA  (RJ2) )  *  (DGAMMA  (RJ3) ) ) 

C  NOW  L  IS  FROM  ZERO  TO  J. AGAIN  CONSIDER  THE  CASE  WHERE  L  IS  ZERO 
LP«=(-1)**J 
PL-LP 

C  NOTE  WHEN  L  IS  ZERO  M  GOES  FROM  ZERO  TO  NT-J 
LJL-NT-J 

IF  (LJL.EQ.O)  GO  TO  2102 
00  2105  M= 1 , L JL 
RRM-M 

RRM1-WNT-RJ+1  .0 
RRM2-WNT-RJ-RRM+1 .0 
RRM3-RRM+1.0 

RCOM* [DGAMMA (RRM1 ) ) /  ( (DGAMMA (RRM2) ) * (DGAMMA  (RRM3) ) ) 

F F  AC*  (DEL**  (-RRM) ) *RCOM 

A-SGM 

B*RRM+SDEL 

CALL  MDBETA (X 1 , A , B , P 1 , I ER) 

CALL  MDBETA  (X2 . A.B, P2 . I ER) 

TA1-A 

TB1«B 

CALL  FACT2  (TA 1 ,TB  1  .TERS) 

Y-TERS 

VALM-  (P2-P1) *FFAC*Y 
VALO-VALO+VALM 
2105  CONTINUE 
2102  RLSUM-VALO*PL 
C  THIS  IS  THE  VALUE  WHEN  L  IS  ZERO 

C  NOW  WANT  TO  CONSIDER  L  FROM  1  TO  J.  THIS  IS  THE  SECOND  SUM 
DO  2501  L-1.J 
RL-L 

RL1-RJ-RL+1.0 
RL2-RL+1 .0 

COMBL-  (DGAMMA  (R J 3) )  /  ( (DGAMMA  (RL 1) )  *  (DGAMMA  (RL2) ) ) 

LPL-  (-1)**  (J-L) 

FLP-LPL 

POWER-DEL** (-RL) 

FACL-FLP*COMBL*POWER 

C  NOW  SHOULD  CONSIDER  M  LOOP  AGAIN. NOW  M  IS  FROM  ZERO  TO  NT-J  FOR  GIVEN  L 
C  START  WITH  MIS  ZERO. 

A-RL+SGM 
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B-SDiL 

CALL  MDBETA  (XI , A,B,P1 , I ER) 
CALL  MDBETA  (X2, A, B,P2, IER) 
TAI-A 


T61-B 

CALL  FACT2 (TA1 ,TB1 ,TERS) 


Y-TERS 

VAL-  (P2-P 1 ) *Y 

RMSUM-VAL 

LL-NT-J 

IF  (LL.EQ.O)  GO  TO  4001 

DO  4000  M-l.LL 

RM-M 

RM1-WNT-RJ+1 .0 
RM2-WNT-RJ-RM+1 .0 
RM3-RM+1 .0 

COMBM- (DGAMMA (RM1) ) / ( (DGAMMA (RM2) ) * (DGAMMA (RM3) ) ) 
F ACM*  (DEL**  (-RM) )  *  (COMBM) 


A-RL+SGM 

B-RM+SDEL 

CALL  MDBETA  (XI, A, B, PI , IER) 

CALL  MDBETA  (X2.A.B.P2, IER) 

TAI-A 

TB1-B 

CALL  FACT2(TA1.TB1.TERS) 

Y-TERS 

VAL-(P2-P1)*FACM*Y 

RMSUM-RMSUM+VAL 

4000  CONTINUE 

4001  RRSUM-RMSUM 

C  THE  MOST  INNER  LOOP  IS  FINISHED. 

RLSUM- (F  ACL*RRSUM) +RLSUM 
C  THIS  IS  THE  SUM  FOR  L  LOOP 
2501  CONTINUE 
C  L  LOOP  IS  FINISHED. 

C  NOW  FINISH  J  LOOP.  THE  MOST  OUTER  LOOP. 

RJSUM- ( COMB J*RL SUM) +RJSUM 
5000  CONTINUE 
C  SO  WE  EVALUATED  G2. 

2011  G2-RJSUM*WW 

WRITE  (6,62)  G2.XT.NT 

62  FORMAT (5X, 'SECOND  CONST- ' F 10 -5»5X , ' XT-' , F5 • 1 ,5X , 
C  CHECK  G2  NOW 

IF  (G2.GE.BET)  GO  TO  777 
XT-XT+1 .0 
GO  TO  888 
999  NT-NT+1 
GO  TO  11 

888  WRITE (6,555)  XT, NT 

555  FORMAT (10X, ' X- 1 , F 10.5i5X* ' N- ' , 1 4) 

STOP 

END 

SUBROUTINE  FACT1  (A1 ,B1 .SON) 

IMPLICIT  REAL*8(A-H,0-Z) 


NT-' ,14) 
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C-A1+B1 

IF  (AT .IE. 57.0. AND. C.LE. 57-0)  GO  TO  U) 

Cl-C-1.0 
A2-A1-1 .0 
B2-B 1-1.0 
C2-A2+B2 
IB-A2+1 .0 
IC-C2 
PAY-C1 

DO  1*2  I-1B, 1C 
21-1 

PAY-PAY*ZI 
1*2  CONTINUE 
PAYDA-1 .0 
JA-B2 

DO  1*3  J-l.JA 
VJ-J 

PAYDA«PAYDA*VJ 
1*3  CONTINUE 

SON-PAY/PAYDA 
GO  TO  1*5 

1*  1  SON-DGAMMA  (C)  /  ( (DGAMrtA  (A  1) )  *  (DGAMMA  (Bl)  ) ) 

1*5  CONTINUE 
RETURN 
END 

SUBROUTINE  F ACT2  (TA 1 ,TB 1 ,TERS) 

IMPLICIT  REAL*8  (A-H.O-Z) 

C-TA1+TB1 

IF  (TA1.LE. 57-0. AND. C.LE. 57.0)  GO  TO  71 

Cl-C-1.0 

A2-TA1-1 .0 

B2-TB 1-1.0 

C2-A2+B2 

IB-A2+1 .0 

IC-C2 

PAY-C1 

DO  72  I - I B , 1C 
Z I  —  I 

PAY-PAY*ZI 

72  CONTINUE 
PAYDA-1 .0 
JA-B2 

DO  73  J-l.JA 
VJ-J 

PAYDA-PAYDA*VJ 

73  CONTINUE 
TERS-PAYDA/PAY 
GO  TO  75 

71  TERS-  ( (DGAnMA  (TAl) )  *  (DGAMMA  (TB1) ) )  /  (DGAKMA  (C) ) 
75  CONTINUE 
RETURN 
END 

//GO.SYSLIB  DD 

//  DD  DSN-GWU. IMSL.V9.DL0AD.DISP-SHR 
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//GO. SYS  IN  DO  * 

0.25000  113.00000  20.00000  0.10100  0.25000  1 

// 


APPENDIX  C 


Illustrative  Calculation  of  Expected  Sample 
Sizes  for  Curtailed  Sequential  Sampling 


C.l.  THE  CASE  OF  TESTING  ONE  ITEM  AT  A  TIME 

We  illustrate  this  for  Stage  0.  Here  x*  =  5  ,  nt  =  17. 

We  must  have  either  6  successes  to  accept,  or  12  failures  to 

reject 


P[nt=6|pt]  = 

P[nt-7 |pt]  = 

P[nt=8|pt]  = 
P[nt=9|pt]  = 
P[nt=10|pt3  = 
P[nt-ll|pt]  = 
P[nt=12|pt]  = 
P[nt=13!pt]  *= 
P[nt=14|pt)  - 


p°  =  0.015625 


V 

r7 ' 
5 


Pt(l-Pt)  =  0. 


046875 


pt(1'pt)2  =  °-0820312 


Pt(1-Pt)  =  °'109375 


p°U-pt)  -  o. 


1230469 


10'  p^(l-pt)5  -  0.1230469 


(?)  *  (n) 

V)  Pjd-P,)7  +  (J2)  PtU-Pt) 

pt(1-pt>8  +  (n)  pt(1-pt) 


fl3l 

5 


12 


12 


0.113037! 

*  0.0968018 

-  0.083313 


34 


35 


P[nt“I5|pt]  *  5*  Pj(l"Pt)9  +  P^(l"Pt)  “  0.0722046 

P|,t-16|pt)  -  I1/)  p^(l-pt)10  +  (“)  p‘(l-Pt)12  -  0.0666504 

P[nt-17jptJ  -  (l6)  p‘(l-Pt)U  +  (J5)  P9(l-Pt>12  -  0.0666504 

To  obtain  P[nt=j]  ,  j  -  6,7, . . .  ,17,  we  average  out  tl 
using  g Cp c  1  * )  •  At  Stage  0,  y  =  1  ,  6  »  1  . 


o  obtain  P[nt=j]  ,  j  -  6,7, .  . .  ,17,  we  average  out  the  above  by 
Cp c | * )  •  At  Stage  0,  y  *  1  ,  6  »  1  . 

P[nt=6]  =  T (y+6+6)  -  0.1428571 

p[\-7J  ' 6  r(m^7)“  '  0-1°n429 

Plnt-0)  '  21  n^EIri  -  °'083333 

PlV9!  '  96  ^^tr1  -  0.0666667 

p'"t'10'  ' 126  "  °-0545455 

ptv11!  '  252  :<?S)-  ’  0-0454545 

,  ,n~  r(Y+6)T(6+6)  r(Y)T(6+12)  _ 

p[nt=12]  =  402  f(Y+j+i2-r  4  r(Y+6+12)  -°-1103896 

r  ...  ...  r(Y+b)T(6+7)  „  r(Y+i)r(6+i2)  _  n 

p[nt=13]  =  792  '  r(Y+5+13)  12  r(Y+6+13)  0.0989011 

r  n/1  1?n7  r(Y+6)T(6+8)  r(Y+2)f(6-H2)  _  OS571/)3 

p[nt=14]  -  1287  r(T+<5+14)  +  78  T(y+6+14)  0.U85714J 

r  ...  „nn„  r(Y+€)r(6+9)  r(Y+3).r_(6+i2i  _  n  n7< 

p["t  155  =  2002  r(Y-H5+15)  36  r  (y+6+15)  U,U/:> 

ptV«>  -  3003  “Sf1  +  1365  ^ 

p[nt«17]  -  4368  +  4368  "  °*0588235 

E[n  ]  -  10.91  . 


36 


37 


p^Cl-Pt)  *  0.0140249 


p["t-»l»ti  •  (3)  pja-pt)4  +  (J)  pja-pt)4  +  (® 

PlV^lft1  •  (3)  pt(1'Pt)4  +  (l3°)  Pt(l-Pt)4  +  (31]  P®(l-Pt)4  +  (9)  p“ 

♦  5°)  pi°a-pt) 


11 

9 


pt°(1'pt)2  “  °*852551 


p[nt=15|pt]  =  p^(l-pt)3  *  0.1291889 

Stage  2 


Pt  =  0.9  (l-pt)  =  0.1 


x*  =  8  n  *  11 


We  must  have  either  9  successes  to  accept  or  3  failures  to  reject, 
Thus,  n  e(3,6,9,12} 


P[nt=3]pt]  = 


(1-p  )  =  0.001 


P[nt=6|pt]  =  2  pt(l-pt)  + 


p[nt=9|pt]  = 


f6l  U(^  ^3  x 

2  Pt(l-Pt)  + 


’41 

2 

pt(1_pt)3  + 

■51 

2 

>  t 

’7' 

5,,  x  3 

f81 

2 

Ptu-Pt)  + 

9 

p3(l-pt)3  •  0.01485 


Putd-Pt) 


3  *  f?l  p9  - 


01  \  ‘  °' 


P [nt*12 1 P 1 3  =  2)  P^1_Pt^2  +  1)  pt^1-pt^  =  °*5596074 
Stage  3 


Pt  =  0.906  l~Pt 


0.094 


x*  -  8  r.fc  -  11 


The  same  enumeration  as  in  Stage  2. 

Stage  4 

Pt  -  0.909  1-p  -  0.091 


x*  ■  8  n  ■  11 


4245426 


The  same  enumeration  as  in  Stage  2. 


38 


Pt  -  0.875  1  -  pt 


0.125 


x*  *  9 


nt  ■  13 


The  same  enumeration  as  in  Stage  1. 


Stage  6 


pt  -  0.853  1  -  pt  =  0.147 


x*  =  8 


12 


We  must  have  either  4  failures  to  reject  or  9  successes  to  accept. 


Thus,  nte{6,9,12} 


Ptnt=6]pt]  = 


4)  PJ(l-pt)4  +  (5]  Pt(l~Pt)5  +  H  (l'Pt)6  -  0.0054577 


P["t-9|pt!  -  (3)  Pjd-Pt)4  +  (3)  p‘d-Pt)4  +  (3)  P,a-Pt)4  +  (0)  Pj  -  0.2653362 


p[nt=12  pt] 


3)  P^d-Pt)3  +  2]  P7t^-Pt)2  +  1]  Ptd“Pt)  -  0.7292061 


Stage  7 


pt  =  0.825  (l-pt)  -  0.175 


x*  «  9 


nt  -  14 


We  must  have  either  10  successes  to  accept  or  5  failures  to  reject. 
Thus,  nte{6,9,12,15} 

p[nt-6lpt]  -  [5]  Pt(l-Pt)5  +  (*]  (1-Pt)6  -  0.0008412 

Plnt*9  J p 1 3  *  4]  P2(l-Pt)5  +  4]  pJO-~Pt)5  +  4  P^1_Pt^5  *  0.0102237 


39 


p[nt*12|pt]  = 
+ 

p[nt=15|pt]  - 


«)  p5t<i-pt>5  +  (“j  pta-pt)5  +  (41)  Pt(1-pt>5  +  (9)  plt 

’1,°]  p“(l-Pt)  +  (Y)  pf  (1-Pt>2  -  0.6805573 

(«2)  Pt(1-Pt>4  +  (p2)  P!<1-Pt>3  '  °-3083778 


Stage  8 


p  =  0.833  (1-p  )  =  0.167 

x*  =  9  nt  =  14 


The  same  enumeration  as  in  Stage  7. 

Stage  9 

p  -  0.820  (l~pt)  =  0.180 

x*  -  9  n  =  14 

The  same  enumeration  as  in  Stage  7. 

Stage  10 

p  =  0.837  (l-pt)  =  0.163 
x*  =  9  nt  -  1A 


The  same  enumeration  as  in  Stage  7 . 

Stage  11 

p  -  0.8A1  (1-p  )  -  0.159 

x*  *  10  nt  "  ^ 


We  must  have  either  11  successes  to  accept  or  5  failures  to  reject 
Thus,  nt  e{6,9, 12,15} 
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Pfnt-6|pt]  -  [*]  pt(l-pt)5  +  [*]  (l-pt)6  -  0.0005289 

p[V9|pt]  =  (*)  P^(l-Pt)5  +  (J)  pJ(l-Pt)5  +  (J)  P^l-Pt)5  -  0.0067523 

p[nt-12|pt]  -=  (*]  P^(l-P£)5  +  (l4°)  P£(l-Pt)5  +  (V)  Pt<l-Pt)5  +  (io)  P 


jj]  P^d-Pj.)  =  0.4321114 


fl2] 

8 .4  . 

fl2^ 

9/,  .3  J 

\l2 

.  1 

.4  J 

Pt(l-Pt)  + 

9  J 

pt^1-pt^  + 

1.10. 

p^°  (l-pt)2  «  0.5606073 


Stage  12 


p  =  0.836  (1-p  )  -  0.164 

x*  =  9  nt  =  14 


The  same  enumeration  as  in  Stage  7. 
Stage  13 


P  -  0.848  (l~p  )  »  0.152 
x*  =  8  nt  =  12 


The  same  enumeration  as  in  Stage  6. 

Stage  14 

p  =  0.850  (l-pt)  *  0.150 

x*  =  8  ■  12 

The  same  enumeration  as  in  Stage  6. 

To  obtain  the  E(nfc)  ,  we  average  out  the  above  by  using  g (p t | • )  . 


We  illustrate  this  for  Stage  0. 


»tv»j  -  Mr  [6  W +  21 1@ri  +  56  IiK»i]  -  °-“7142’ 

r  ,01  m+6)  r(v+6)T(6+4)  f(Y+6)r(6+5)_  402  r(Y^)r(646) 

p[nt*12]  «  r(Y)T(5)  L126  T(y+5+10)  25  T(y+<5+11)  r(Y+<S+12) 

r(Y)T(  64122.1  =  0.2103897 
T(Y+6+12)  J 

,  r(Y+<5)  fi--  rfY-^>r(Y+7)  .  r(Y+6)r(6+8)  4  IlxtllXiiiil 

plnt=15]  *  r(Y)r<6)  L92  r(Y+<5+13)  1287  T(y+<S+14)  r(Y+^+15) 

„  r(Y+i)r(«+i2)  .  no  r(Y+2_ir (6+121  j.  =  n.?sqf,is4 

+  12  rfY+S+13)  f(Y+6+14)  +  T(y+6+15)  J 

r  --  r(Y+5)  finm  T(Y+5)r (6+10)  6  r  (y+4)  r  ( 6+1 1_).  1  =  0.125 

p[nt-18]  =  r(Y)r(^)-  j _ 3003  r(Y+6+15)  1365  r(Y+3+15)  J 


(.2103897 


E[n  ]  =  11.84  . 

Similarly,  we  can  obtain  Efn^]  for  other  stages. 
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